2024-03-25T16:22:03.353 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:22:03.353 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275 --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/orig.config.yaml 2024-03-25T16:22:03.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:22:03.492 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621275' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOqIqfEQZt615P3doFwq9RpYUjamDcNMxl4wqu0jpH8yYxlrfRlZ9eQhZSGpvnHoa8mNuD8n7zg2fH01SeUZWE= smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCi6NsqnuXO4iVWJhs8L6iaW9zhmGjeZxY8mKhUJZvmHnPsroUzqgUn1tmOoUIKd0cSSaEEofFq8BfyG5kZ8ksI= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:22:03.493 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:22:03.493 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:22:03.494 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:22:03.496 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:22:03.523 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:22:03.523 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:22:03.523 INFO:teuthology.packaging:ref: None 2024-03-25T16:22:03.524 INFO:teuthology.packaging:tag: None 2024-03-25T16:22:03.524 INFO:teuthology.packaging:branch: main 2024-03-25T16:22:03.524 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:22:03.524 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:22:03.753 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:22:03.754 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:22:03.761 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:22:03.761 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:22:03.770 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:22:03.787 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:22:03.804 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:22:03.830 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:17:15.246194', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOqIqfEQZt615P3doFwq9RpYUjamDcNMxl4wqu0jpH8yYxlrfRlZ9eQhZSGpvnHoa8mNuD8n7zg2fH01SeUZWE='} 2024-03-25T16:22:03.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:17:15.247707', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCi6NsqnuXO4iVWJhs8L6iaW9zhmGjeZxY8mKhUJZvmHnPsroUzqgUn1tmOoUIKd0cSSaEEofFq8BfyG5kZ8ksI='} 2024-03-25T16:22:03.855 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:22:03.866 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:22:03.867 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:22:03.867 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:22:03.946 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f10809fd0d0>, signals=[15]) 2024-03-25T16:22:03.946 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:22:03.957 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:22:03.957 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T16:22:03.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:04.065 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T16:22:04.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:04.170 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:22:04.181 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2024-03-25T16:22:04.200 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2024-03-25T16:22:04.200 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2024-03-25T16:22:04.257 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2024-03-25T16:22:04.257 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="9" 2024-03-25T16:22:04.257 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2024-03-25T16:22:04.257 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="9" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:22:04.258 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:22:04.259 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2024-03-25T16:22:04.282 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-03-25T16:22:04.301 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-03-25T16:22:04.301 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-03-25T16:22:04.358 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-03-25T16:22:04.358 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-03-25T16:22:04.358 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:22:04.359 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:22:04.359 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-03-25T16:22:04.382 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:22:04.393 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:22:04.409 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:22:04.409 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:22:04.411 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:22:04.428 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:22:04.437 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:22:04.437 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:22:04.469 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:22:04.498 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:22:04.556 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:22:04.572 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:22:04.573 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:22:04.573 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:22:04.573 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:22:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:22:04.591 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-25T16:22:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:22:04.592 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-25T16:22:04.648 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:22:04.648 INFO:teuthology.task.kernel:Running kernel on smithi022: 5.14.0-361.el9.x86_64 2024-03-25T16:22:04.649 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2024-03-25T16:22:04.650 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:22:04.651 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2024-03-25T16:22:04.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-03-25T16:22:06.928 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:22:07.049 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T16:22:08.487 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-25T16:22:08.633 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-25T16:22:09.498 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-25T16:22:11.860 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-03-25T16:22:12.524 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 19 MB 00:01 2024-03-25T16:22:23.832 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 814 kB/s | 6.1 MB 00:07 2024-03-25T16:22:24.893 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 1.6 MB/s | 19 MB 00:11 2024-03-25T16:22:25.482 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T16:22:27.445 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:22:29.043 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:22:30.695 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-25T16:22:32.525 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-25T16:22:32.620 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:22:34.757 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:22:34.757 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:22:34.849 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:22:34.863 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:22:34.863 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-25T16:22:34.863 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:22:34.863 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:22:34.863 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2024-03-25T16:22:34.864 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:22:34.865 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 76 M 2024-03-25T16:22:34.865 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 123 M 2024-03-25T16:22:34.865 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:22:35.118 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 392 kB/s | 44 kB 00:00 2024-03-25T16:22:35.669 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-25T16:22:36.210 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-25T16:22:36.886 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-25T16:22:36.886 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:36.886 INFO:teuthology.orchestra.run.smithi033.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-25T16:22:37.577 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:22:37.720 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:22:37.996 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:22:37.996 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:22:39.061 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:22:39.062 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:22:39.944 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:22:39.944 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:22:40.039 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:22:40.054 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:22:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-25T16:22:40.054 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:22:40.054 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:22:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:22:40.054 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:22:40.054 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout:Install 4 Packages 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 76 M 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 123 M 2024-03-25T16:22:40.055 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:22:40.575 INFO:teuthology.orchestra.run.smithi022.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 93 kB/s | 44 kB 00:00 2024-03-25T16:22:41.666 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:22:42.323 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:22:42.410 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:45.294 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:22:45.480 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:48.421 INFO:teuthology.orchestra.run.smithi022.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 1.8 MB/s | 15 MB 00:08 2024-03-25T16:22:50.552 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:22:50.656 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:51.071 INFO:teuthology.orchestra.run.smithi022.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 2.6 MB/s | 27 MB 00:10 2024-03-25T16:22:53.997 INFO:teuthology.orchestra.run.smithi022.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 2.4 MB/s | 33 MB 00:13 2024-03-25T16:22:53.997 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:53.997 INFO:teuthology.orchestra.run.smithi022.stdout:Total 5.4 MB/s | 76 MB 00:13 2024-03-25T16:22:54.693 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:22:55.111 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:22:55.751 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:22:56.170 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:22:56.171 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:22:58.828 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:22:59.450 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:22:59.538 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:23:02.427 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:23:02.565 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:23:07.648 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:23:07.693 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:12.786 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:26.920 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:26.921 INFO:teuthology.orchestra.run.smithi033.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:23:26.921 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:23:26.967 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:27.339 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:27.339 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:23:27.339 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:23:27.339 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:23:27.750 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:23:28.092 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:23:28.205 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:23:28.206 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:23:28.206 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:23:28.206 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:23:28.206 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:23:28.207 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:23:28.207 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:23:28.207 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:23:28.290 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:23:28.290 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:23:28.290 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:23:28.339 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:23:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:23:28.341 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:23:28.341 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:23:28.341 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:23:28.902 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-03-25T16:23:30.257 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:23:30.295 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-03-25T16:23:30.297 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-25T16:23:30.312 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.Me9TYOJoYA 2024-03-25T16:23:30.312 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Me9TYOJoYA 2024-03-25T16:23:30.384 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.Me9TYOJoYA 2024-03-25T16:23:30.529 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.Me9TYOJoYA is 6KB 2024-03-25T16:23:30.578 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.Me9TYOJoYA 2024-03-25T16:23:30.593 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:23:30.670 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:23:30.671 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:23:30.671 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:23:30.671 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:23:30.672 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:23:30.835 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-03-25T16:23:43.345 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:43.346 INFO:teuthology.orchestra.run.smithi022.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:23:43.346 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:23:43.392 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:23:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:23:43.813 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:23:44.238 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:23:44.238 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:23:44.239 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:23:44.239 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:23:44.239 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:23:44.239 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:23:44.239 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:23:44.239 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:23:44.239 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:23:44.554 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:23:44.656 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:23:44.656 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:23:44.656 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:23:44.657 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:23:44.657 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:23:44.657 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:23:44.657 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:23:44.658 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:23:44.741 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:23:44.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:23:44.742 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2024-03-25T16:23:44.781 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.06 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 68.el9 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Group : Unspecified 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 8264254 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:23:44.782 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : builder@centos.org 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : CentOS 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:23:44.783 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:23:44.784 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:23:44.784 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:23:44.784 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:23:45.363 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2024-03-25T16:23:46.803 INFO:teuthology.orchestra.run.smithi022.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:23:46.842 INFO:teuthology.orchestra.run.smithi022.stderr:done 2024-03-25T16:23:46.844 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-25T16:23:46.860 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.MRCepFtI69 2024-03-25T16:23:46.860 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MRCepFtI69 2024-03-25T16:23:46.932 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.MRCepFtI69 2024-03-25T16:23:47.072 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.MRCepFtI69 is 6KB 2024-03-25T16:23:47.120 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.MRCepFtI69 2024-03-25T16:23:47.135 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:23:47.209 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:23:47.209 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:23:47.209 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:23:47.209 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:23:47.214 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:23:47.365 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2024-03-25T16:24:00.850 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:24:00.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-25T16:24:00.852 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:17.382 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:24:17.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-25T16:24:17.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:19.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-25T16:24:28.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-25T16:24:28.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:35.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.22 2024-03-25T16:24:44.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-25T16:24:44.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:24:46.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-03-25T16:24:58.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-25T16:24:58.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:25:00.511 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-25T16:25:01.119 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-25T16:25:01.119 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:25:01.120 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-03-25T16:25:01.135 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:25:01.135 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:25:01.135 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:25:01.135 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:25:02.136 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:25:02.136 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:25:02.298 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-03-25T16:25:02.344 DEBUG:teuthology.parallel:result is None 2024-03-25T16:25:44.878 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:25:56.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-25T16:25:56.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:25:57.260 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-25T16:25:57.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-25T16:25:57.669 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:25:57.670 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-25T16:25:57.688 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:25:57.689 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:25:57.689 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:25:57.689 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:25:58.690 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:25:58.690 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:25:58.759 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2024-03-25T16:25:58.793 DEBUG:teuthology.parallel:result is None 2024-03-25T16:25:58.794 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:25:58.811 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:25:58.811 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:25:58.813 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:25:58.833 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:25:58.839 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:25:58.845 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:25:58.845 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:25:58.872 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:25:58.901 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:25:58.907 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:25:58.908 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:25:58.932 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:25:58.971 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:58.978 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:58.984 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:58.993 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:25:58.995 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:25:59.003 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:25:59.003 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:25:59.029 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:25:59.065 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:25:59.075 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:25:59.076 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:25:59.099 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:25:59.122 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:59.197 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:59.246 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:59.325 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:59.372 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T16:25:59.372 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:25:59.439 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:59.483 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:25:59.536 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:59.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:25:59.664 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T16:25:59.664 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:25:59.732 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2024-03-25T16:25:59.734 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-03-25T16:25:59.775 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:25:59.827 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:26:00.302 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:26:00.311 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:26:00.311 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:26:00.321 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:26:00.331 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2024-03-25T16:26:00.387 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: 2024-03-25T16:26:00.389 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-03-25T16:26:00.461 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-03-25T16:26:00.463 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:26:00.463 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2024-03-25T16:26:00.489 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2024-03-25T16:26:00.489 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-03-25T16:26:00.514 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-03-25T16:26:00.514 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi033.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:26:00.514 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:26:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:26:00.560 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:26:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:26:00.590 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:26:00.590 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:26:00.631 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:26:00.666 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:26:00.675 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:26:00.714 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:26:00.716 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi033.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:30:19.239 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi033.front.sepia.ceph.com')] 2024-03-25T16:30:19.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-25T16:30:19.270 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:30:19.367 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-25T16:30:19.453 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-25T16:30:19.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-25T16:30:19.454 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:30:19.544 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-03-25T16:30:19.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-03-25T16:30:19.637 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:30:19.647 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:30:19.648 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:30:19.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:30:19.650 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:30:19.651 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:30:19.679 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:30:19.696 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:30:19.708 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:30:19.724 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:30:19.725 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2024-03-25T16:30:19.739 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2024-03-25T16:30:19.753 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-03-25T16:30:19.755 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:30:19.767 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-03-25T16:30:19.772 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:30:19.783 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:30:19.799 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:30:19.825 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:30:19.829 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:30:19.829 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-25T16:30:19.829 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.829 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.829 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.830 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.849 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:30:19.853 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:30:19.853 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-25T16:30:19.853 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.853 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.853 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.853 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:30:19.854 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:30:19.862 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:30:19.862 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:30:19.862 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:30:19.862 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:30:19.868 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:30:19.868 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:30:19.869 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:30:19.869 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:30:19.871 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:30:19.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:30:20.023 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:30:20.024 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:30:20.226 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:30:20.226 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:30:20.365 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:30:20.366 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T16:30:20.366 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:30:20.396 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:30:20.396 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:30:20.465 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:30:20.546 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:30:20.557 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:30:20.558 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T16:30:20.558 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:30:20.586 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:30:20.586 DEBUG:teuthology.orchestra.run.smithi033:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:30:20.612 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2024-03-25T16:30:20.614 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2024-03-25T16:30:20.653 DEBUG:teuthology.orchestra.run.smithi033:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:30:20.733 DEBUG:teuthology.orchestra.run.smithi033:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:30:20.798 INFO:teuthology.orchestra.run.smithi033.stdout:check_obsoletes = 1 2024-03-25T16:30:20.800 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum clean all 2024-03-25T16:30:20.855 INFO:teuthology.orchestra.run.smithi022.stdout:74 files removed 2024-03-25T16:30:20.883 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw 2024-03-25T16:30:21.047 INFO:teuthology.orchestra.run.smithi033.stdout:74 files removed 2024-03-25T16:30:21.075 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw 2024-03-25T16:30:22.307 INFO:teuthology.orchestra.run.smithi033.stdout:ceph packages for x86_64 88 kB/s | 82 kB 00:00 2024-03-25T16:30:22.723 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 53 kB/s | 82 kB 00:01 2024-03-25T16:30:23.594 INFO:teuthology.orchestra.run.smithi033.stdout:ceph noarch packages 8.1 kB/s | 10 kB 00:01 2024-03-25T16:30:23.624 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:30:24.441 INFO:teuthology.orchestra.run.smithi033.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:30:24.449 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:30:24.893 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T16:30:25.319 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.7 MB 00:00 2024-03-25T16:30:26.360 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-25T16:30:26.785 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-25T16:30:29.176 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-25T16:30:29.442 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-03-25T16:30:45.618 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-25T16:30:51.560 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.1 MB 00:02 2024-03-25T16:30:52.257 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 888 kB/s | 19 MB 00:21 2024-03-25T16:30:53.085 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-25T16:30:54.481 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T16:30:56.391 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-25T16:30:57.792 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 85 kB/s | 16 kB 00:00 2024-03-25T16:30:59.210 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T16:30:59.504 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T16:31:02.054 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:31:02.070 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:31:02.070 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrading: 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:31:02.071 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse 2024-03-25T16:31:02.072 INFO:teuthology.orchestra.run.smithi033.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout:Installing weak dependencies: 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:31:02.073 INFO:teuthology.orchestra.run.smithi033.stdout:Install 30 Packages 2024-03-25T16:31:02.074 INFO:teuthology.orchestra.run.smithi033.stdout:Upgrade 2 Packages 2024-03-25T16:31:02.074 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:31:02.074 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 58 M 2024-03-25T16:31:02.074 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:31:04.335 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-25T16:31:06.857 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:31:06.876 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:31:06.877 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:31:06.878 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout:Installing weak dependencies: 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout:Install 30 Packages 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 2 Packages 2024-03-25T16:31:06.879 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:06.880 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 58 M 2024-03-25T16:31:06.880 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:31:09.453 INFO:teuthology.orchestra.run.smithi022.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.2 MB/s | 9.7 MB 00:02 2024-03-25T16:31:09.503 INFO:teuthology.orchestra.run.smithi022.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 505 kB/s | 25 kB 00:00 2024-03-25T16:31:09.670 INFO:teuthology.orchestra.run.smithi022.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.4 MB/s | 742 kB 00:00 2024-03-25T16:31:09.770 INFO:teuthology.orchestra.run.smithi022.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-25T16:31:10.779 INFO:teuthology.orchestra.run.smithi022.stdout:(5/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 5.4 MB/s | 20 MB 00:03 2024-03-25T16:31:10.846 INFO:teuthology.orchestra.run.smithi022.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 4.7 MB/s | 5.1 MB 00:01 2024-03-25T16:31:11.521 INFO:teuthology.orchestra.run.smithi022.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 61 kB/s | 45 kB 00:00 2024-03-25T16:31:11.588 INFO:teuthology.orchestra.run.smithi022.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 177 kB/s | 131 kB 00:00 2024-03-25T16:31:11.896 INFO:teuthology.orchestra.run.smithi022.stdout:(9/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 5.3 MB 00:04 2024-03-25T16:31:12.155 INFO:teuthology.orchestra.run.smithi022.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 565 kB/s | 320 kB 00:00 2024-03-25T16:31:12.255 INFO:teuthology.orchestra.run.smithi022.stdout:(11/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 221 kB/s | 162 kB 00:00 2024-03-25T16:31:12.305 INFO:teuthology.orchestra.run.smithi022.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 729 kB/s | 298 kB 00:00 2024-03-25T16:31:12.355 INFO:teuthology.orchestra.run.smithi022.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 496 kB/s | 99 kB 00:00 2024-03-25T16:31:12.573 INFO:teuthology.orchestra.run.smithi022.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 204 kB/s | 44 kB 00:00 2024-03-25T16:31:12.597 INFO:teuthology.orchestra.run.smithi022.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 309 kB/s | 106 kB 00:00 2024-03-25T16:31:12.648 INFO:teuthology.orchestra.run.smithi022.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 507 kB/s | 159 kB 00:00 2024-03-25T16:31:12.815 INFO:teuthology.orchestra.run.smithi022.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.7 MB/s | 661 kB 00:00 2024-03-25T16:31:12.840 INFO:teuthology.orchestra.run.smithi022.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.1 MB/s | 282 kB 00:00 2024-03-25T16:31:12.865 INFO:teuthology.orchestra.run.smithi022.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 190 kB/s | 41 kB 00:00 2024-03-25T16:31:13.182 INFO:teuthology.orchestra.run.smithi022.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 141 kB/s | 44 kB 00:00 2024-03-25T16:31:13.241 INFO:teuthology.orchestra.run.smithi022.stdout:(21/32): qatlib-service-23.11.0-1.el9.x86_64.rp 87 kB/s | 35 kB 00:00 2024-03-25T16:31:13.266 INFO:teuthology.orchestra.run.smithi022.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 481 kB/s | 216 kB 00:00 2024-03-25T16:31:13.850 INFO:teuthology.orchestra.run.smithi022.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 44 kB/s | 26 kB 00:00 2024-03-25T16:31:14.083 INFO:teuthology.orchestra.run.smithi022.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 208 kB/s | 49 kB 00:00 2024-03-25T16:31:14.192 INFO:teuthology.orchestra.run.smithi022.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 306 kB/s | 309 kB 00:01 2024-03-25T16:31:14.317 INFO:teuthology.orchestra.run.smithi022.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 288 kB/s | 67 kB 00:00 2024-03-25T16:31:14.651 INFO:teuthology.orchestra.run.smithi022.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 575 kB/s | 192 kB 00:00 2024-03-25T16:31:15.618 INFO:teuthology.orchestra.run.smithi022.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 588 kB/s | 838 kB 00:01 2024-03-25T16:31:17.259 INFO:teuthology.orchestra.run.smithi022.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 621 kB/s | 1.6 MB 00:02 2024-03-25T16:31:20.501 INFO:teuthology.orchestra.run.smithi022.stdout:(30/32): libarrow-9.0.0-10.el9.x86_64.rpm 623 kB/s | 4.4 MB 00:07 2024-03-25T16:31:20.744 INFO:teuthology.orchestra.run.smithi033.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 303 kB/s | 5.3 MB 00:18 2024-03-25T16:31:20.928 INFO:teuthology.orchestra.run.smithi033.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 137 kB/s | 25 kB 00:00 2024-03-25T16:31:21.384 INFO:teuthology.orchestra.run.smithi022.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 599 kB/s | 3.4 MB 00:05 2024-03-25T16:31:21.601 INFO:teuthology.orchestra.run.smithi022.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 738 kB/s | 3.1 MB 00:04 2024-03-25T16:31:21.612 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:21.612 INFO:teuthology.orchestra.run.smithi022.stdout:Total 3.9 MB/s | 58 MB 00:14 2024-03-25T16:31:21.645 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:31:21.686 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:31:21.687 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:31:22.183 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:31:22.183 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:31:22.944 INFO:teuthology.orchestra.run.smithi033.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 368 kB/s | 742 kB 00:02 2024-03-25T16:31:23.694 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:31:23.787 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:31:23.814 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:31:23.847 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:31:23.878 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:23.895 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:23.919 INFO:teuthology.orchestra.run.smithi033.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 511 kB/s | 498 kB 00:00 2024-03-25T16:31:24.085 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:24.146 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:31:24.532 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:24.625 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:24.715 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:31:24.742 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:24.804 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:24.833 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:31:24.878 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:31:24.947 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:31:24.970 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:25.040 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:25.068 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:31:25.180 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:31:25.219 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:31:25.266 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:31:25.317 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:31:25.346 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:31:25.379 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:31:25.707 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:31:25.733 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:25.815 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:25.846 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:31:25.886 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:31:26.362 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:31:26.386 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:26.457 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:26.528 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:31:26.563 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:31:28.813 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:28.842 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:29.410 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:29.454 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:29.536 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:29.537 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:31:29.537 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:31:29.537 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:29.571 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:33.061 INFO:teuthology.orchestra.run.smithi033.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 329 kB/s | 9.7 MB 00:30 2024-03-25T16:31:33.227 INFO:teuthology.orchestra.run.smithi033.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 271 kB/s | 45 kB 00:00 2024-03-25T16:31:33.644 INFO:teuthology.orchestra.run.smithi033.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 315 kB/s | 131 kB 00:00 2024-03-25T16:31:34.219 INFO:teuthology.orchestra.run.smithi033.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 281 kB/s | 162 kB 00:00 2024-03-25T16:31:35.420 INFO:teuthology.orchestra.run.smithi033.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 267 kB/s | 320 kB 00:01 2024-03-25T16:31:36.370 INFO:teuthology.orchestra.run.smithi033.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 313 kB/s | 298 kB 00:00 2024-03-25T16:31:37.028 INFO:teuthology.orchestra.run.smithi033.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 151 kB/s | 99 kB 00:00 2024-03-25T16:31:37.104 INFO:teuthology.orchestra.run.smithi033.stdout:(12/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 396 kB/s | 5.1 MB 00:13 2024-03-25T16:31:37.230 INFO:teuthology.orchestra.run.smithi033.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 530 kB/s | 106 kB 00:00 2024-03-25T16:31:37.254 INFO:teuthology.orchestra.run.smithi033.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T16:31:37.279 INFO:teuthology.orchestra.run.smithi033.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-25T16:31:37.330 INFO:teuthology.orchestra.run.smithi033.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T16:31:37.355 INFO:teuthology.orchestra.run.smithi033.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T16:31:37.381 INFO:teuthology.orchestra.run.smithi033.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 820 kB/s | 41 kB 00:00 2024-03-25T16:31:37.506 INFO:teuthology.orchestra.run.smithi033.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 279 kB/s | 35 kB 00:00 2024-03-25T16:31:37.598 INFO:teuthology.orchestra.run.smithi033.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 486 kB/s | 44 kB 00:00 2024-03-25T16:31:37.623 INFO:teuthology.orchestra.run.smithi033.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 810 kB/s | 216 kB 00:00 2024-03-25T16:31:38.132 INFO:teuthology.orchestra.run.smithi033.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 579 kB/s | 309 kB 00:00 2024-03-25T16:31:38.274 INFO:teuthology.orchestra.run.smithi033.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-03-25T16:31:38.357 INFO:teuthology.orchestra.run.smithi033.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 592 kB/s | 49 kB 00:00 2024-03-25T16:31:38.441 INFO:teuthology.orchestra.run.smithi033.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 820 kB/s | 67 kB 00:00 2024-03-25T16:31:38.500 INFO:teuthology.orchestra.run.smithi033.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.0 MB/s | 4.4 MB 00:00 2024-03-25T16:31:38.625 INFO:teuthology.orchestra.run.smithi033.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-25T16:31:38.650 INFO:teuthology.orchestra.run.smithi033.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-25T16:31:38.767 INFO:teuthology.orchestra.run.smithi033.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /root 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2024-03-25T16:31:39.133 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:40.064 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:31:40.089 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:31:40.089 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:40.089 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:31:40.089 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:31:40.089 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:31:40.090 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:40.091 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:31:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:31:40.175 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:41.402 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:41.402 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:31:41.403 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:31:41.404 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:31:41.405 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:31:41.405 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:31:41.405 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:31:41.405 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:31:41.405 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:31:41.405 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:31:42.601 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.602 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.603 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:31:42.604 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:42.605 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:31:43.005 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-test 2024-03-25T16:31:43.576 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:39 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:31:43.738 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:31:43.739 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:31:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:43.740 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:31:43.740 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:31:43.740 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-25T16:31:43.740 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:31:43.740 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 48 M 2024-03-25T16:31:43.740 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 204 M 2024-03-25T16:31:43.741 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:31:43.960 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T16:31:44.011 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:31:44.044 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:31:44.236 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-25T16:31:44.903 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 146 kB/s | 163 kB 00:01 2024-03-25T16:31:48.216 INFO:teuthology.orchestra.run.smithi033.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 361 kB/s | 3.4 MB 00:09 2024-03-25T16:31:49.508 INFO:teuthology.orchestra.run.smithi033.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 298 kB/s | 3.1 MB 00:10 2024-03-25T16:31:51.432 INFO:teuthology.orchestra.run.smithi033.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 418 kB/s | 20 MB 00:48 2024-03-25T16:31:51.438 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:51.438 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.2 MB/s | 58 MB 00:49 2024-03-25T16:31:51.469 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:31:51.511 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:31:51.551 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:31:52.005 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:31:52.005 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:31:53.481 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:31:53.565 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:31:53.596 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:31:53.643 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:31:53.669 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:53.686 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:31:53.889 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:53.942 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:31:54.286 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:31:54.340 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:54.416 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:31:54.506 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:31:54.533 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:54.579 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:31:54.608 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:31:54.652 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:31:54.722 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:31:54.744 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:54.798 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:31:54.827 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:31:54.938 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:31:54.977 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:31:55.018 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:31:55.059 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:31:55.104 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:31:55.137 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:31:55.479 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:31:55.509 INFO:teuthology.orchestra.run.smithi033.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:55.582 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:31:55.613 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:31:55.646 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:31:56.130 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:31:56.161 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:56.241 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:31:56.312 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:31:56.355 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:31:58.660 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:58.701 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:59.277 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:31:59.314 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:59.414 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:31:59.414 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:31:59.414 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:31:59.414 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:31:59.456 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:32:09.084 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /sys 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /proc 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /mnt 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /var/tmp 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /home 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /root 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout:skipping the directory /tmp 2024-03-25T16:32:09.085 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:32:10.019 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:32:10.046 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:32:10.046 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:32:10.046 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:32:10.046 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:32:10.046 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:32:10.046 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:32:10.048 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:32:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:32:10.130 INFO:teuthology.orchestra.run.smithi033.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:32:11.184 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:32:11.184 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:32:11.184 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:32:11.184 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:32:11.184 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:32:11.184 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:32:11.184 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:32:11.185 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:32:11.186 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:32:11.187 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:32:11.187 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:32:11.187 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:32:11.187 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:32:11.187 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:32:11.187 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:32:12.303 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout:Upgraded: 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.304 INFO:teuthology.orchestra.run.smithi033.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.305 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:12.306 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:32:12.306 INFO:teuthology.orchestra.run.smithi033.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:32:12.306 INFO:teuthology.orchestra.run.smithi033.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:32:12.306 INFO:teuthology.orchestra.run.smithi033.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:32:12.306 INFO:teuthology.orchestra.run.smithi033.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:32:12.306 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:32:12.306 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:32:12.566 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-test 2024-03-25T16:32:13.137 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:01:14 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:32:13.290 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:32:13.291 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:32:13.291 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:32:13.291 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:32:13.291 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:32:13.291 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:32:13.291 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:32:13.291 INFO:teuthology.orchestra.run.smithi033.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-03-25T16:32:13.292 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:32:13.293 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 48 M 2024-03-25T16:32:13.293 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 204 M 2024-03-25T16:32:13.293 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:32:13.590 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T16:32:13.640 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-25T16:32:13.665 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T16:32:13.849 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-25T16:32:14.416 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 164 kB/s | 163 kB 00:00 2024-03-25T16:33:23.757 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 484 kB/s | 47 MB 01:39 2024-03-25T16:33:23.758 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:23.759 INFO:teuthology.orchestra.run.smithi022.stdout:Total 493 kB/s | 48 MB 01:40 2024-03-25T16:33:23.769 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:33:23.790 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:33:23.790 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:33:24.181 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:33:24.182 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:33:24.791 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:33:24.857 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:33:24.894 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:33:24.934 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:33:24.982 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:33:25.009 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:33:29.712 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:33:29.794 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:33:32.091 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:33:32.091 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:33:32.091 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:33:32.091 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:33:32.091 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:33:32.091 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:33:32.465 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:33:32.465 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:33:32.466 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:33:32.798 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph 2024-03-25T16:33:33.376 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:29 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:33:33.541 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:33:33.543 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:33:33.544 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:33:33.545 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:33:33.546 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:33:33.547 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:33:33.547 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:33:33.547 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-25T16:33:33.547 INFO:teuthology.orchestra.run.smithi022.stdout:Install 36 Packages 2024-03-25T16:33:33.547 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:33:33.547 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 30 M 2024-03-25T16:33:33.547 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 107 M 2024-03-25T16:33:33.548 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:33:34.493 INFO:teuthology.orchestra.run.smithi022.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:33:37.851 INFO:teuthology.orchestra.run.smithi022.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 409 kB/s | 1.5 MB 00:03 2024-03-25T16:33:40.701 INFO:teuthology.orchestra.run.smithi022.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 330 kB/s | 2.1 MB 00:06 2024-03-25T16:33:41.369 INFO:teuthology.orchestra.run.smithi022.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 371 kB/s | 247 kB 00:00 2024-03-25T16:33:41.477 INFO:teuthology.orchestra.run.smithi022.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-25T16:33:41.519 INFO:teuthology.orchestra.run.smithi022.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:33:41.552 INFO:teuthology.orchestra.run.smithi022.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 947 kB/s | 32 kB 00:00 2024-03-25T16:33:41.586 INFO:teuthology.orchestra.run.smithi022.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:33:41.619 INFO:teuthology.orchestra.run.smithi022.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:33:41.662 INFO:teuthology.orchestra.run.smithi022.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:33:41.737 INFO:teuthology.orchestra.run.smithi022.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:33:41.771 INFO:teuthology.orchestra.run.smithi022.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:33:41.804 INFO:teuthology.orchestra.run.smithi022.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:33:41.838 INFO:teuthology.orchestra.run.smithi022.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:33:41.930 INFO:teuthology.orchestra.run.smithi022.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-03-25T16:33:42.082 INFO:teuthology.orchestra.run.smithi022.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 149 kB/s | 22 kB 00:00 2024-03-25T16:33:42.505 INFO:teuthology.orchestra.run.smithi022.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 355 kB/s | 151 kB 00:00 2024-03-25T16:33:42.847 INFO:teuthology.orchestra.run.smithi022.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 127 kB/s | 43 kB 00:00 2024-03-25T16:33:43.272 INFO:teuthology.orchestra.run.smithi022.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 404 kB/s | 172 kB 00:00 2024-03-25T16:33:43.864 INFO:teuthology.orchestra.run.smithi022.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 606 kB/s | 358 kB 00:00 2024-03-25T16:33:44.064 INFO:teuthology.orchestra.run.smithi022.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 53 kB/s | 11 kB 00:00 2024-03-25T16:33:44.331 INFO:teuthology.orchestra.run.smithi022.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-25T16:33:44.598 INFO:teuthology.orchestra.run.smithi022.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-03-25T16:33:44.857 INFO:teuthology.orchestra.run.smithi022.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 75 kB/s | 19 kB 00:00 2024-03-25T16:33:45.124 INFO:teuthology.orchestra.run.smithi022.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 74 kB/s | 20 kB 00:00 2024-03-25T16:33:45.474 INFO:teuthology.orchestra.run.smithi022.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 132 kB/s | 46 kB 00:00 2024-03-25T16:33:45.824 INFO:teuthology.orchestra.run.smithi022.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 225 kB/s | 79 kB 00:00 2024-03-25T16:33:46.324 INFO:teuthology.orchestra.run.smithi022.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 545 kB/s | 272 kB 00:00 2024-03-25T16:33:46.591 INFO:teuthology.orchestra.run.smithi022.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 62 kB/s | 16 kB 00:00 2024-03-25T16:33:46.933 INFO:teuthology.orchestra.run.smithi022.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 263 kB/s | 90 kB 00:00 2024-03-25T16:33:47.200 INFO:teuthology.orchestra.run.smithi022.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-25T16:33:47.708 INFO:teuthology.orchestra.run.smithi022.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 452 kB/s | 230 kB 00:00 2024-03-25T16:33:48.234 INFO:teuthology.orchestra.run.smithi022.stdout:(33/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 349 kB/s | 4.7 MB 00:13 2024-03-25T16:33:48.293 INFO:teuthology.orchestra.run.smithi022.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 731 kB/s | 427 kB 00:00 2024-03-25T16:33:48.510 INFO:teuthology.orchestra.run.smithi022.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 73 kB/s | 20 kB 00:00 2024-03-25T16:34:18.499 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 387 kB/s | 47 MB 02:05 2024-03-25T16:34:18.501 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:18.501 INFO:teuthology.orchestra.run.smithi033.stdout:Total 394 kB/s | 48 MB 02:05 2024-03-25T16:34:18.512 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:34:18.532 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:34:18.533 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:34:18.926 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:34:18.926 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:34:19.532 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:34:19.608 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:34:19.651 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:34:19.683 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:34:19.732 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:34:19.760 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:34:24.489 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:34:24.561 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:34:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:34:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:34:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:34:26.984 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:34:26.985 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:34:26.985 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:34:27.389 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:34:27.389 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:34:27.390 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:34:27.701 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph 2024-03-25T16:34:28.269 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:34:28.438 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:34:28.441 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:34:28.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:34:28.443 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout:========================================================================================= 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout:Install 36 Packages 2024-03-25T16:34:28.444 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:34:28.445 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 30 M 2024-03-25T16:34:28.445 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 107 M 2024-03-25T16:34:28.445 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:34:29.117 INFO:teuthology.orchestra.run.smithi033.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:34:31.459 INFO:teuthology.orchestra.run.smithi033.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 562 kB/s | 1.5 MB 00:02 2024-03-25T16:34:33.551 INFO:teuthology.orchestra.run.smithi033.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 452 kB/s | 2.1 MB 00:04 2024-03-25T16:34:34.142 INFO:teuthology.orchestra.run.smithi033.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 418 kB/s | 247 kB 00:00 2024-03-25T16:34:34.251 INFO:teuthology.orchestra.run.smithi033.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-25T16:34:34.285 INFO:teuthology.orchestra.run.smithi033.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T16:34:34.318 INFO:teuthology.orchestra.run.smithi033.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-25T16:34:34.351 INFO:teuthology.orchestra.run.smithi033.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:34:34.385 INFO:teuthology.orchestra.run.smithi033.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:34:34.436 INFO:teuthology.orchestra.run.smithi033.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:34:34.519 INFO:teuthology.orchestra.run.smithi033.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T16:34:34.553 INFO:teuthology.orchestra.run.smithi033.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:34:34.586 INFO:teuthology.orchestra.run.smithi033.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T16:34:34.628 INFO:teuthology.orchestra.run.smithi033.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:34:34.704 INFO:teuthology.orchestra.run.smithi033.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-03-25T16:34:34.754 INFO:teuthology.orchestra.run.smithi033.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-03-25T16:34:35.296 INFO:teuthology.orchestra.run.smithi033.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 279 kB/s | 151 kB 00:00 2024-03-25T16:34:35.413 INFO:teuthology.orchestra.run.smithi033.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-03-25T16:34:35.571 INFO:teuthology.orchestra.run.smithi033.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-25T16:34:35.772 INFO:teuthology.orchestra.run.smithi033.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T16:34:35.872 INFO:teuthology.orchestra.run.smithi033.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T16:34:35.972 INFO:teuthology.orchestra.run.smithi033.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2024-03-25T16:34:36.364 INFO:teuthology.orchestra.run.smithi033.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 59 kB/s | 23 kB 00:00 2024-03-25T16:34:36.456 INFO:teuthology.orchestra.run.smithi033.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-25T16:34:36.548 INFO:teuthology.orchestra.run.smithi033.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-03-25T16:34:36.648 INFO:teuthology.orchestra.run.smithi033.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T16:34:36.757 INFO:teuthology.orchestra.run.smithi033.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 728 kB/s | 79 kB 00:00 2024-03-25T16:34:36.882 INFO:teuthology.orchestra.run.smithi033.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T16:34:36.974 INFO:teuthology.orchestra.run.smithi033.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-03-25T16:34:37.083 INFO:teuthology.orchestra.run.smithi033.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 832 kB/s | 90 kB 00:00 2024-03-25T16:34:37.183 INFO:teuthology.orchestra.run.smithi033.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 358 kB/s | 36 kB 00:00 2024-03-25T16:34:37.258 INFO:teuthology.orchestra.run.smithi033.stdout:(32/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 589 kB/s | 4.7 MB 00:08 2024-03-25T16:34:37.300 INFO:teuthology.orchestra.run.smithi033.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-25T16:34:37.400 INFO:teuthology.orchestra.run.smithi033.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T16:34:37.901 INFO:teuthology.orchestra.run.smithi033.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 665 kB/s | 427 kB 00:00 2024-03-25T16:34:47.485 INFO:teuthology.orchestra.run.smithi022.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 247 kB/s | 17 MB 01:09 2024-03-25T16:34:47.488 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:47.488 INFO:teuthology.orchestra.run.smithi022.stdout:Total 411 kB/s | 30 MB 01:13 2024-03-25T16:34:47.512 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:34:47.557 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:34:47.557 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:34:47.969 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:34:47.969 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:34:49.343 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:34:49.395 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:34:49.430 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:34:49.464 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:34:49.538 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:34:49.580 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:34:49.613 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:34:49.708 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:34:49.785 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:34:49.849 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:34:49.924 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:34:49.984 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:34:50.022 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:34:50.079 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:34:50.143 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:34:50.334 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:34:50.376 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:34:50.414 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:34:50.457 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:34:50.487 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:34:50.566 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:34:50.597 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:34:50.631 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:34:50.672 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:34:50.715 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:34:50.749 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:34:50.842 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:34:50.880 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:34:50.955 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:34:51.103 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:34:51.229 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:34:51.395 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:34:51.426 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:34:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:34:51.451 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:51.451 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:34:51.451 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:34:51.451 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:34:51.451 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:52.968 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:34:52.994 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:34:52.994 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:52.995 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:34:52.995 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:34:52.995 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:34:52.995 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:53.461 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:34:53.484 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:34:53.484 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:53.484 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:34:53.484 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:34:53.484 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:34:53.485 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:53.695 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:34:53.717 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:34:53.717 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:53.718 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:34:53.718 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:34:53.718 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:34:53.718 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:53.777 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:34:55.414 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:34:55.415 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:34:55.416 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:34:55.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:34:55.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:34:55.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:34:55.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:34:55.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:34:56.499 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:34:56.500 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:34:56.501 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:56.502 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:34:56.871 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base 2024-03-25T16:34:57.446 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:53 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:34:57.546 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:57.601 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:34:57.602 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:34:57.602 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:34:57.677 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install cephadm 2024-03-25T16:34:58.247 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:34:58.402 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:34:58.403 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:34:58.403 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 747 k 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 754 k 2024-03-25T16:34:58.404 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:35:03.352 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 151 kB/s | 747 kB 00:04 2024-03-25T16:35:03.353 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:03.354 INFO:teuthology.orchestra.run.smithi022.stdout:Total 151 kB/s | 747 kB 00:04 2024-03-25T16:35:03.354 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:35:03.360 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:35:03.360 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:35:03.367 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:35:03.367 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:35:03.445 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:35:03.915 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:04.010 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:04.386 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:04.676 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:04.676 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:04.676 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:35:04.677 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:04.677 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:04.677 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:35:04.894 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:35:05.180 INFO:teuthology.orchestra.run.smithi033.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 509 kB/s | 17 MB 00:33 2024-03-25T16:35:05.186 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:05.186 INFO:teuthology.orchestra.run.smithi033.stdout:Total 827 kB/s | 30 MB 00:36 2024-03-25T16:35:05.207 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:35:05.252 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:35:05.252 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:35:05.473 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:35:05.629 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:35:05.630 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:35:05.630 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T16:35:05.630 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 145 k 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 436 k 2024-03-25T16:35:05.631 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:35:05.665 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:35:05.666 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:35:06.942 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 111 kB/s | 145 kB 00:01 2024-03-25T16:35:06.943 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:06.943 INFO:teuthology.orchestra.run.smithi022.stdout:Total 111 kB/s | 145 kB 00:01 2024-03-25T16:35:06.943 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:35:06.953 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:35:06.953 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:35:06.988 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:35:07.011 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:35:07.011 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:35:07.050 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:35:07.085 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:35:07.119 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:35:07.153 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:35:07.194 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:35:07.218 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:07.242 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:07.243 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:07.243 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:35:07.243 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:07.243 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:35:07.292 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:35:07.388 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:35:07.473 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:35:07.521 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:35:07.594 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:35:07.655 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:35:07.692 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:35:07.751 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:35:07.831 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:35:08.025 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:35:08.072 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:35:08.110 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:35:08.144 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:35:08.174 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:35:08.207 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:08.208 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:08.208 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:35:08.208 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:08.208 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:08.208 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:35:08.246 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:35:08.276 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:35:08.309 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:35:08.350 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:35:08.385 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:35:08.428 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:35:08.514 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr 2024-03-25T16:35:08.522 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:35:08.559 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:35:08.634 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:35:08.776 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:35:08.906 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:35:09.075 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:35:09.090 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:35:09.105 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:35:09.129 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:35:09.129 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:09.129 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:35:09.129 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:35:09.129 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:35:09.129 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:09.190 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:09.244 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:35:09.245 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:35:09.245 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:35:09.320 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:35:09.889 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:35:10.044 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:35:10.045 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:35:10.046 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout:Install 12 Packages 2024-03-25T16:35:10.047 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:10.048 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.9 M 2024-03-25T16:35:10.048 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2024-03-25T16:35:10.048 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:35:10.667 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:35:10.693 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:35:10.693 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:10.693 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:35:10.693 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:35:10.693 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:35:10.693 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:10.913 INFO:teuthology.orchestra.run.smithi022.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-25T16:35:11.062 INFO:teuthology.orchestra.run.smithi022.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T16:35:11.096 INFO:teuthology.orchestra.run.smithi022.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 7.0 MB/s | 238 kB 00:00 2024-03-25T16:35:11.182 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:35:11.196 INFO:teuthology.orchestra.run.smithi022.stdout:(4/12): ceph-grafana-dashboards-19.0.0-1439.g5d 43 kB/s | 31 kB 00:00 2024-03-25T16:35:11.204 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:35:11.204 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:11.204 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:35:11.204 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:35:11.205 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:35:11.205 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:11.413 INFO:teuthology.orchestra.run.smithi022.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-25T16:35:11.424 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:35:11.446 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:35:11.447 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:11.447 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:35:11.447 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:35:11.447 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:35:11.447 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:11.516 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:35:11.589 INFO:teuthology.orchestra.run.smithi022.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-03-25T16:35:11.980 INFO:teuthology.orchestra.run.smithi022.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.2 MB/s | 2.0 MB 00:00 2024-03-25T16:35:12.064 INFO:teuthology.orchestra.run.smithi022.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-25T16:35:12.142 INFO:teuthology.orchestra.run.smithi022.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 410 kB/s | 31 kB 00:00 2024-03-25T16:35:12.223 INFO:teuthology.orchestra.run.smithi022.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T16:35:13.211 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:35:13.212 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:35:13.213 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:35:13.215 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:35:14.450 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:35:14.450 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:14.450 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:35:14.451 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:35:14.452 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:14.453 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:35:14.631 INFO:teuthology.orchestra.run.smithi022.stdout:(11/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 160 kB/s | 548 kB 00:03 2024-03-25T16:35:14.796 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-base 2024-03-25T16:35:15.382 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:16 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:35:15.484 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:15.539 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:35:15.540 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:35:15.540 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:35:15.615 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install cephadm 2024-03-25T16:35:16.186 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:35:16.341 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:35:16.342 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:35:16.342 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:35:16.342 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:35:16.342 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:35:16.342 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 747 k 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 754 k 2024-03-25T16:35:16.343 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:35:19.131 INFO:teuthology.orchestra.run.smithi022.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 211 kB/s | 1.8 MB 00:08 2024-03-25T16:35:19.134 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:19.134 INFO:teuthology.orchestra.run.smithi022.stdout:Total 773 kB/s | 6.9 MB 00:09 2024-03-25T16:35:19.155 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:35:19.174 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:35:19.174 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:35:19.360 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:35:19.361 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:35:19.911 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:35:19.990 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:35:20.025 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:35:20.072 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:35:20.102 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:35:20.212 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:35:20.305 INFO:teuthology.orchestra.run.smithi033.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 189 kB/s | 747 kB 00:03 2024-03-25T16:35:20.305 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:20.306 INFO:teuthology.orchestra.run.smithi033.stdout:Total 189 kB/s | 747 kB 00:03 2024-03-25T16:35:20.306 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:35:20.313 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:35:20.313 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:35:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:35:20.319 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:35:20.349 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:35:20.399 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:35:20.443 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:35:20.491 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:35:20.527 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:35:20.559 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:35:20.814 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:35:20.842 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:35:20.876 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:20.971 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:21.364 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:35:21.628 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:35:21.629 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:35:21.629 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:35:21.629 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:35:21.629 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:35:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:35:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:21.632 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:35:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:21.632 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:21.632 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:35:21.876 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:35:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:35:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:21.989 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:35:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:35:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:21.990 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:35:22.210 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:35:22.453 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:23 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:35:22.609 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:35:22.610 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:35:22.611 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:35:22.611 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:22.611 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 145 k 2024-03-25T16:35:22.611 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 436 k 2024-03-25T16:35:22.611 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:35:22.785 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:35:22.941 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:35:22.942 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:35:22.943 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout:Install 12 Packages 2024-03-25T16:35:22.944 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:35:22.945 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 37 M 2024-03-25T16:35:22.945 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 209 M 2024-03-25T16:35:22.945 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:35:23.430 INFO:teuthology.orchestra.run.smithi022.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 195 kB/s | 32 kB 00:00 2024-03-25T16:35:23.463 INFO:teuthology.orchestra.run.smithi022.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 519 kB/s | 17 kB 00:00 2024-03-25T16:35:23.496 INFO:teuthology.orchestra.run.smithi022.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:35:23.672 INFO:teuthology.orchestra.run.smithi022.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T16:35:23.805 INFO:teuthology.orchestra.run.smithi022.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:35:23.836 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 119 kB/s | 145 kB 00:01 2024-03-25T16:35:23.836 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:23.837 INFO:teuthology.orchestra.run.smithi033.stdout:Total 118 kB/s | 145 kB 00:01 2024-03-25T16:35:23.837 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:35:23.847 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:35:23.847 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:35:23.847 INFO:teuthology.orchestra.run.smithi022.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T16:35:23.906 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:35:23.906 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:35:23.906 INFO:teuthology.orchestra.run.smithi022.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2024-03-25T16:35:24.058 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:35:24.073 INFO:teuthology.orchestra.run.smithi022.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2024-03-25T16:35:24.136 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:24.161 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:24.161 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:35:24.161 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:35:24.161 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:24.224 INFO:teuthology.orchestra.run.smithi022.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-25T16:35:24.249 INFO:teuthology.orchestra.run.smithi022.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-25T16:35:25.128 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:35:25.128 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:25.128 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:35:25.128 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:35:25.128 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:25.128 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:35:25.191 INFO:teuthology.orchestra.run.smithi022.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 254 kB/s | 245 kB 00:00 2024-03-25T16:35:25.389 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr 2024-03-25T16:35:25.964 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:26 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:35:26.064 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:35:26.118 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:35:26.119 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:35:26.120 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:35:26.187 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:35:26.756 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:35:26.911 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:35:26.912 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-25T16:35:26.912 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:35:26.912 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-25T16:35:26.912 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:35:26.912 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:26.913 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:35:26.914 INFO:teuthology.orchestra.run.smithi033.stdout:====================================================================================== 2024-03-25T16:35:26.914 INFO:teuthology.orchestra.run.smithi033.stdout:Install 12 Packages 2024-03-25T16:35:26.914 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:26.914 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 6.9 M 2024-03-25T16:35:26.914 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 26 M 2024-03-25T16:35:26.914 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:35:27.664 INFO:teuthology.orchestra.run.smithi033.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T16:35:27.806 INFO:teuthology.orchestra.run.smithi033.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-03-25T16:35:27.848 INFO:teuthology.orchestra.run.smithi033.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 5.6 MB/s | 238 kB 00:00 2024-03-25T16:35:28.132 INFO:teuthology.orchestra.run.smithi033.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-25T16:35:28.157 INFO:teuthology.orchestra.run.smithi033.stdout:(5/12): ceph-grafana-dashboards-19.0.0-1439.g5d 30 kB/s | 31 kB 00:01 2024-03-25T16:35:28.407 INFO:teuthology.orchestra.run.smithi033.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-03-25T16:35:28.591 INFO:teuthology.orchestra.run.smithi033.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T16:35:28.699 INFO:teuthology.orchestra.run.smithi033.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.0 MB/s | 2.0 MB 00:00 2024-03-25T16:35:28.725 INFO:teuthology.orchestra.run.smithi033.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T16:35:28.767 INFO:teuthology.orchestra.run.smithi033.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 460 kB/s | 31 kB 00:00 2024-03-25T16:35:28.792 INFO:teuthology.orchestra.run.smithi033.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:35:34.575 INFO:teuthology.orchestra.run.smithi033.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 244 kB/s | 1.8 MB 00:07 2024-03-25T16:35:34.580 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:34.580 INFO:teuthology.orchestra.run.smithi033.stdout:Total 916 kB/s | 6.9 MB 00:07 2024-03-25T16:35:34.600 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:35:34.620 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:35:34.620 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:35:34.799 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:35:34.800 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:35:35.317 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:35:35.401 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:35:35.436 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:35:35.483 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:35:35.521 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:35:35.625 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:35:35.760 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:35:35.852 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:35:35.893 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:35:35.920 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:35:35.952 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:35:36.208 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:35:36.235 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:35:37.011 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:35:37.362 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:35:37.363 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:37.364 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:35:37.607 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:35:38.181 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:35:38.337 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:35:38.338 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-25T16:35:38.338 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:35:38.338 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-25T16:35:38.338 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:35:38.338 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:38.339 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:35:38.340 INFO:teuthology.orchestra.run.smithi033.stdout:============================================================================================ 2024-03-25T16:35:38.340 INFO:teuthology.orchestra.run.smithi033.stdout:Install 12 Packages 2024-03-25T16:35:38.340 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:35:38.340 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 37 M 2024-03-25T16:35:38.340 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 209 M 2024-03-25T16:35:38.340 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:35:38.823 INFO:teuthology.orchestra.run.smithi033.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 239 kB/s | 32 kB 00:00 2024-03-25T16:35:38.881 INFO:teuthology.orchestra.run.smithi033.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 505 kB/s | 17 kB 00:00 2024-03-25T16:35:38.906 INFO:teuthology.orchestra.run.smithi033.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:35:39.065 INFO:teuthology.orchestra.run.smithi033.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T16:35:39.199 INFO:teuthology.orchestra.run.smithi033.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:35:39.249 INFO:teuthology.orchestra.run.smithi033.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.3 MB/s | 3.0 MB 00:00 2024-03-25T16:35:39.274 INFO:teuthology.orchestra.run.smithi033.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-25T16:35:39.633 INFO:teuthology.orchestra.run.smithi033.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-25T16:35:40.466 INFO:teuthology.orchestra.run.smithi033.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 234 kB/s | 195 kB 00:00 2024-03-25T16:35:40.583 INFO:teuthology.orchestra.run.smithi033.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-25T16:35:41.091 INFO:teuthology.orchestra.run.smithi033.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 442 kB/s | 803 kB 00:01 2024-03-25T16:35:52.396 INFO:teuthology.orchestra.run.smithi022.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 260 kB/s | 7.4 MB 00:29 2024-03-25T16:35:52.399 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:35:52.399 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.3 MB/s | 37 MB 00:29 2024-03-25T16:35:52.442 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:35:52.468 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:35:52.468 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:35:52.842 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:35:52.842 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:35:53.588 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:35:53.652 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:35:53.683 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:35:53.792 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:35:53.828 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:35:54.013 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:35:54.044 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:35:54.130 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:35:54.816 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:35:54.944 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:35:56.240 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:35:57.318 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:35:57.343 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:35:59.760 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:35:59.761 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:35:59.761 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:35:59.761 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:35:59.761 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:36:00.144 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:36:00.145 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:36:00.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:36:00.146 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:36:00.146 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:00.146 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:00.504 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:36:01.085 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:04:57 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:01.241 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:01.242 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T16:36:01.242 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:36:01.242 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T16:36:01.242 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:01.242 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:36:01.242 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:36:01.243 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:01.244 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:01.244 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T16:36:01.244 INFO:teuthology.orchestra.run.smithi022.stdout:Install 13 Packages 2024-03-25T16:36:01.244 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:01.244 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 2.0 M 2024-03-25T16:36:01.245 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2024-03-25T16:36:01.245 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:01.372 INFO:teuthology.orchestra.run.smithi033.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 334 kB/s | 7.4 MB 00:22 2024-03-25T16:36:01.375 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:01.375 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.6 MB/s | 37 MB 00:23 2024-03-25T16:36:01.417 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:01.444 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:01.444 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:01.460 INFO:teuthology.orchestra.run.smithi022.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 107 kB/s | 18 kB 00:00 2024-03-25T16:36:01.485 INFO:teuthology.orchestra.run.smithi022.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 125 kB/s | 24 kB 00:00 2024-03-25T16:36:01.544 INFO:teuthology.orchestra.run.smithi022.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-25T16:36:01.569 INFO:teuthology.orchestra.run.smithi022.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-25T16:36:01.602 INFO:teuthology.orchestra.run.smithi022.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:36:01.620 INFO:teuthology.orchestra.run.smithi022.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-25T16:36:01.815 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:01.816 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:01.853 INFO:teuthology.orchestra.run.smithi022.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 128 kB/s | 32 kB 00:00 2024-03-25T16:36:01.870 INFO:teuthology.orchestra.run.smithi022.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-25T16:36:02.029 INFO:teuthology.orchestra.run.smithi022.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.1 MB/s | 203 kB 00:00 2024-03-25T16:36:02.054 INFO:teuthology.orchestra.run.smithi022.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 68 kB/s | 52 kB 00:00 2024-03-25T16:36:02.162 INFO:teuthology.orchestra.run.smithi022.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-03-25T16:36:02.179 INFO:teuthology.orchestra.run.smithi022.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 394 kB/s | 59 kB 00:00 2024-03-25T16:36:02.204 INFO:teuthology.orchestra.run.smithi022.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 596 kB/s | 90 kB 00:00 2024-03-25T16:36:02.209 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:02.209 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-25T16:36:02.209 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:02.224 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:02.224 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:02.320 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:02.322 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:02.687 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:02.762 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:36:02.793 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:36:02.843 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:02.892 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:36:02.930 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:36:03.020 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:36:03.082 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:36:03.115 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:36:03.138 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:36:03.145 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:36:03.173 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:36:03.215 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:36:03.229 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:36:03.314 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:36:03.432 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:36:03.485 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:36:03.920 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:36:04.046 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:36:04.251 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:36:04.305 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:36:04.339 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:36:04.388 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:36:04.417 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:36:05.191 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:36:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:36:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:36:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:36:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:36:05.192 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:36:05.348 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:36:05.560 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:36:05.561 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:36:05.561 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:36:05.561 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:36:05.561 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:05.561 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:05.815 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:36:06.390 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:06.432 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:36:06.461 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:36:06.549 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:06.549 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T16:36:06.549 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:06.550 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T16:36:06.551 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-25T16:36:06.551 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:06.551 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.8 M 2024-03-25T16:36:06.551 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 33 M 2024-03-25T16:36:06.552 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:07.440 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 556 kB/s | 248 kB 00:00 2024-03-25T16:36:07.532 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-25T16:36:07.640 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 213 kB/s | 138 kB 00:00 2024-03-25T16:36:07.682 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2024-03-25T16:36:07.799 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-25T16:36:07.824 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 277 kB/s | 51 kB 00:00 2024-03-25T16:36:07.827 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:07.827 INFO:teuthology.orchestra.run.smithi022.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-25T16:36:07.840 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:07.851 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:07.851 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:07.943 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:07.945 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:08.244 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:08.421 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:36:08.468 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:36:08.961 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:36:08.961 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:36:08.961 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:36:08.961 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:36:08.961 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:36:08.962 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:36:08.962 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:36:08.962 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:36:08.962 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:36:08.962 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:36:08.962 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:36:08.962 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:36:09.029 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:36:09.123 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:36:09.192 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:36:09.222 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:36:09.441 INFO:teuthology.orchestra.run.smithi033.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:09.442 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:09.879 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:36:10.004 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:36:10.004 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:36:10.004 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:36:10.005 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:36:10.005 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:36:10.005 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:10.315 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:10.456 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:10.581 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-fuse 2024-03-25T16:36:10.612 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:36:10.613 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T16:36:10.614 INFO:teuthology.orchestra.run.smithi033.stdout:Install 13 Packages 2024-03-25T16:36:10.615 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:10.615 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 2.0 M 2024-03-25T16:36:10.615 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 26 M 2024-03-25T16:36:10.615 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:10.928 INFO:teuthology.orchestra.run.smithi033.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 188 kB/s | 24 kB 00:00 2024-03-25T16:36:10.953 INFO:teuthology.orchestra.run.smithi033.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-25T16:36:11.003 INFO:teuthology.orchestra.run.smithi033.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:36:11.028 INFO:teuthology.orchestra.run.smithi033.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:36:11.054 INFO:teuthology.orchestra.run.smithi033.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:36:11.079 INFO:teuthology.orchestra.run.smithi033.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-25T16:36:11.159 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:11.313 INFO:teuthology.orchestra.run.smithi033.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 101 kB/s | 52 kB 00:00 2024-03-25T16:36:11.315 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:11.316 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:11.317 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-25T16:36:11.317 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:11.317 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 964 k 2024-03-25T16:36:11.317 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 2.8 M 2024-03-25T16:36:11.317 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:11.338 INFO:teuthology.orchestra.run.smithi033.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-25T16:36:11.363 INFO:teuthology.orchestra.run.smithi033.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-03-25T16:36:11.418 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 782 kB/s | 79 kB 00:00 2024-03-25T16:36:11.672 INFO:teuthology.orchestra.run.smithi033.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 565 kB/s | 203 kB 00:00 2024-03-25T16:36:11.697 INFO:teuthology.orchestra.run.smithi033.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 177 kB/s | 59 kB 00:00 2024-03-25T16:36:11.797 INFO:teuthology.orchestra.run.smithi033.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 717 kB/s | 90 kB 00:00 2024-03-25T16:36:11.831 INFO:teuthology.orchestra.run.smithi033.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.1 MB/s | 1.0 MB 00:00 2024-03-25T16:36:11.833 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:11.833 INFO:teuthology.orchestra.run.smithi033.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-25T16:36:11.833 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:11.849 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:11.849 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:11.951 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:11.952 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:12.484 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:12.660 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:36:12.716 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:36:12.772 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:36:12.808 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:36:12.850 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:36:12.946 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:36:13.057 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:36:13.119 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:36:13.880 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:36:13.922 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:36:13.956 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:36:14.005 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:36:14.018 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 328 kB/s | 885 kB 00:02 2024-03-25T16:36:14.019 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:14.020 INFO:teuthology.orchestra.run.smithi022.stdout:Total 357 kB/s | 964 kB 00:02 2024-03-25T16:36:14.020 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:14.043 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:36:14.044 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:14.044 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:14.123 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:14.124 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:14.307 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:14.440 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:36:14.521 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:36:14.793 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:36:14.793 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:36:14.793 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:36:14.793 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:36:14.794 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:36:15.163 INFO:teuthology.orchestra.run.smithi033.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:15.164 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:15.197 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:36:15.197 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:36:15.423 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:36:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:36:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:15.505 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:36:15.505 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:15.505 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:15.799 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-volume 2024-03-25T16:36:15.996 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:16.153 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:16.154 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:16.155 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T16:36:16.155 INFO:teuthology.orchestra.run.smithi033.stdout:Install 6 Packages 2024-03-25T16:36:16.155 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:16.155 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 6.8 M 2024-03-25T16:36:16.155 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 33 M 2024-03-25T16:36:16.155 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:16.382 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:16.538 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:16.539 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:16.539 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 332 k 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.5 M 2024-03-25T16:36:16.540 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:16.549 INFO:teuthology.orchestra.run.smithi033.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T16:36:16.574 INFO:teuthology.orchestra.run.smithi033.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 829 kB/s | 248 kB 00:00 2024-03-25T16:36:16.641 INFO:teuthology.orchestra.run.smithi033.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 865 kB/s | 58 kB 00:00 2024-03-25T16:36:16.647 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 644 kB/s | 69 kB 00:00 2024-03-25T16:36:16.674 INFO:teuthology.orchestra.run.smithi033.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.5 MB/s | 51 kB 00:00 2024-03-25T16:36:16.708 INFO:teuthology.orchestra.run.smithi033.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.4 MB/s | 548 kB 00:00 2024-03-25T16:36:17.041 INFO:teuthology.orchestra.run.smithi033.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 180 kB/s | 138 kB 00:00 2024-03-25T16:36:17.042 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:17.043 INFO:teuthology.orchestra.run.smithi033.stdout:Total 7.7 MB/s | 6.8 MB 00:00 2024-03-25T16:36:17.057 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:17.068 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:17.068 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:17.162 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:17.162 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:17.467 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:17.667 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:36:17.719 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:36:18.039 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 176 kB/s | 264 kB 00:01 2024-03-25T16:36:18.040 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:18.041 INFO:teuthology.orchestra.run.smithi022.stdout:Total 221 kB/s | 332 kB 00:01 2024-03-25T16:36:18.041 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:18.052 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:18.052 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:18.094 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:18.094 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:18.248 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:18.273 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:36:18.348 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:36:18.417 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:36:18.434 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:36:18.447 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:36:18.475 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:36:18.490 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:36:18.490 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:18.490 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:36:18.490 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:36:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:36:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:36:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:36:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:36:19.209 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:36:19.284 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:19.515 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:19.621 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:36:19.621 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:19.621 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:19.621 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:19.622 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:36:19.622 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:19.622 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:19.744 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-fuse 2024-03-25T16:36:19.962 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados-devel 2024-03-25T16:36:20.323 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:20.479 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:36:20.480 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:20.481 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:20.481 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:20.481 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-03-25T16:36:20.481 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:20.481 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 964 k 2024-03-25T16:36:20.481 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 2.8 M 2024-03-25T16:36:20.481 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:20.542 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:20.603 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 649 kB/s | 79 kB 00:00 2024-03-25T16:36:20.698 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:36:20.699 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:20.700 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 127 k 2024-03-25T16:36:20.700 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 456 k 2024-03-25T16:36:20.700 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:21.388 INFO:teuthology.orchestra.run.smithi022.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 184 kB/s | 127 kB 00:00 2024-03-25T16:36:21.389 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:21.389 INFO:teuthology.orchestra.run.smithi022.stdout:Total 183 kB/s | 127 kB 00:00 2024-03-25T16:36:21.389 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:21.396 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:21.396 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:21.456 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:21.456 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:21.606 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:21.707 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:22.176 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:22.480 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:22.480 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:22.480 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:22.480 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:22.481 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:22.481 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:22.715 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs2 2024-03-25T16:36:23.294 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:23.394 INFO:teuthology.orchestra.run.smithi022.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:23.451 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:23.452 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:36:23.452 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:23.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs-devel 2024-03-25T16:36:24.101 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:24.255 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:24.256 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:36:24.257 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:24.257 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 31 k 2024-03-25T16:36:24.257 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 140 k 2024-03-25T16:36:24.257 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:24.802 INFO:teuthology.orchestra.run.smithi022.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 57 kB/s | 31 kB 00:00 2024-03-25T16:36:24.802 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:24.802 INFO:teuthology.orchestra.run.smithi022.stdout:Total 57 kB/s | 31 kB 00:00 2024-03-25T16:36:24.802 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:24.804 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:24.804 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:24.828 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:24.828 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:24.961 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:25.011 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 195 kB/s | 885 kB 00:04 2024-03-25T16:36:25.013 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:25.013 INFO:teuthology.orchestra.run.smithi033.stdout:Total 213 kB/s | 964 kB 00:04 2024-03-25T16:36:25.013 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:25.038 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:25.038 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:25.067 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:25.118 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:25.118 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:25.304 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:25.447 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:25.466 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:36:25.538 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:36:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:25.722 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:25.722 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:25.722 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:25.858 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados2 2024-03-25T16:36:26.203 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:36:26.203 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:36:26.436 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:22 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:26.496 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:36:26.496 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:26.496 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:26.496 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:36:26.496 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:26.496 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:26.536 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:26.590 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:26.592 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:36:26.592 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:26.643 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librbd1 2024-03-25T16:36:26.748 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-volume 2024-03-25T16:36:27.223 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:27.323 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:28 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:27.328 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:27.383 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:27.384 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:36:27.384 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:27.443 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rados 2024-03-25T16:36:27.480 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:27.481 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-03-25T16:36:27.481 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:27.481 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:36:27.481 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:36:27.481 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:36:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:27.482 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:27.482 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:27.482 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-03-25T16:36:27.482 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:27.482 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 332 k 2024-03-25T16:36:27.482 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.5 M 2024-03-25T16:36:27.483 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:27.598 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 593 kB/s | 69 kB 00:00 2024-03-25T16:36:28.028 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:28.129 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:28.184 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:28.185 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:36:28.185 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:28.234 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rgw 2024-03-25T16:36:28.582 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 240 kB/s | 264 kB 00:01 2024-03-25T16:36:28.583 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:28.584 INFO:teuthology.orchestra.run.smithi033.stdout:Total 302 kB/s | 332 kB 00:01 2024-03-25T16:36:28.584 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:28.594 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:28.594 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:28.642 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:28.643 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:28.802 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:28.810 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:28.910 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:28.965 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:28.966 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:36:28.966 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:28.981 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:36:29.016 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-cephfs 2024-03-25T16:36:29.017 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:36:29.033 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:36:29.033 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:29.033 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:36:29.033 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:29.595 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:29.695 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:29.749 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:29.750 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:36:29.750 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:29.795 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:36:29.803 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rbd 2024-03-25T16:36:30.137 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:36:30.137 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:30.137 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:30.137 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:36:30.137 INFO:teuthology.orchestra.run.smithi033.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:36:30.137 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:30.137 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:30.377 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:26 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:30.464 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados-devel 2024-03-25T16:36:30.475 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:30.530 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:30.531 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T16:36:30.531 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:30.586 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-fuse 2024-03-25T16:36:31.046 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:31.161 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:27 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:31.202 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:31.203 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 127 k 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 456 k 2024-03-25T16:36:31.204 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:31.315 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:31.316 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:31.316 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-25T16:36:31.316 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:31.316 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:31.316 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:36:31.316 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:31.317 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:31.317 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:31.317 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:36:31.317 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:31.317 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 85 k 2024-03-25T16:36:31.317 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 231 k 2024-03-25T16:36:31.317 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:32.095 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 110 kB/s | 85 kB 00:00 2024-03-25T16:36:32.095 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:32.095 INFO:teuthology.orchestra.run.smithi022.stdout:Total 109 kB/s | 85 kB 00:00 2024-03-25T16:36:32.096 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:32.104 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:32.104 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:32.166 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:32.166 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:32.312 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:32.329 INFO:teuthology.orchestra.run.smithi033.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 113 kB/s | 127 kB 00:01 2024-03-25T16:36:32.330 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:32.330 INFO:teuthology.orchestra.run.smithi033.stdout:Total 112 kB/s | 127 kB 00:01 2024-03-25T16:36:32.331 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:32.338 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:32.338 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:32.402 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:32.402 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:32.413 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:32.553 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:32.655 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:32.815 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:33.090 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:33.102 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:33.103 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:33.103 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:33.103 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:33.337 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-mirror 2024-03-25T16:36:33.344 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:33.345 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:33.345 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:33.345 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:33.345 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:33.345 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:33.579 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs2 2024-03-25T16:36:33.912 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:29 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:34.068 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:34.069 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:36:34.070 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:34.070 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 3.1 M 2024-03-25T16:36:34.070 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 13 M 2024-03-25T16:36:34.070 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:34.157 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:34.259 INFO:teuthology.orchestra.run.smithi033.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:34.316 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:34.317 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:36:34.317 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:34.387 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs-devel 2024-03-25T16:36:34.964 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:35.118 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:36:35.119 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:35.120 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 31 k 2024-03-25T16:36:35.120 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 140 k 2024-03-25T16:36:35.120 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:35.865 INFO:teuthology.orchestra.run.smithi033.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 42 kB/s | 31 kB 00:00 2024-03-25T16:36:35.866 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:35.866 INFO:teuthology.orchestra.run.smithi033.stdout:Total 42 kB/s | 31 kB 00:00 2024-03-25T16:36:35.866 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:35.870 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:35.870 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:35.900 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:35.900 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:36.040 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:36.160 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:36.514 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:36.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:36.776 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:36.776 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:36.777 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:36.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:36.777 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:36.950 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados2 2024-03-25T16:36:37.539 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:38 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:37.643 INFO:teuthology.orchestra.run.smithi033.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:37.699 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:37.700 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:36:37.701 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:37.753 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librbd1 2024-03-25T16:36:38.189 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 767 kB/s | 3.1 MB 00:04 2024-03-25T16:36:38.190 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:38.190 INFO:teuthology.orchestra.run.smithi022.stdout:Total 767 kB/s | 3.1 MB 00:04 2024-03-25T16:36:38.191 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:38.202 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:38.202 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:38.282 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:38.282 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:38.337 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:38.437 INFO:teuthology.orchestra.run.smithi033.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:38.493 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:38.494 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:36:38.494 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:38.544 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rados 2024-03-25T16:36:38.740 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:38.806 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:38.830 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:38.831 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:38.831 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:36:38.831 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:36:38.831 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:36:38.831 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:39.120 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:39.220 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:39.277 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:39.278 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:36:39.278 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:39.325 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rgw 2024-03-25T16:36:39.899 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:40 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:39.968 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:39.969 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:39.969 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:39.969 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:39.998 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:40.052 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:40.053 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:36:40.053 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:40.100 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-cephfs 2024-03-25T16:36:40.233 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-nbd 2024-03-25T16:36:40.673 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:40.771 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:40.814 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:40.825 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:40.827 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:36:40.827 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:40.885 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rbd 2024-03-25T16:36:40.970 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:40.970 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:36:40.971 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:40.972 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 171 k 2024-03-25T16:36:40.972 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 494 k 2024-03-25T16:36:40.972 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:41.460 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:42 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:41.559 INFO:teuthology.orchestra.run.smithi033.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:36:41.613 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:41.615 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T16:36:41.615 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:41.665 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-fuse 2024-03-25T16:36:41.984 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 169 kB/s | 171 kB 00:01 2024-03-25T16:36:41.985 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:41.985 INFO:teuthology.orchestra.run.smithi022.stdout:Total 169 kB/s | 171 kB 00:01 2024-03-25T16:36:41.986 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:36:41.996 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:36:41.996 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:36:42.059 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:36:42.060 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:36:42.212 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:36:42.245 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:43 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:42.319 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:42.402 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:42.402 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 85 k 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 231 k 2024-03-25T16:36:42.403 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:42.729 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:43.029 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:43.030 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:36:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:43.030 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:43.030 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:36:43.185 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 109 kB/s | 85 kB 00:00 2024-03-25T16:36:43.186 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:43.186 INFO:teuthology.orchestra.run.smithi033.stdout:Total 109 kB/s | 85 kB 00:00 2024-03-25T16:36:43.186 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:43.194 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:43.194 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:43.252 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:36:43.254 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:43.255 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:43.395 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:43.487 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:43.829 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:05:39 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:36:43.861 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:43.986 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:36:43.987 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:43.988 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:36:43.988 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:36:43.988 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-25T16:36:43.988 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:36:43.988 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 91 M 2024-03-25T16:36:43.988 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 498 M 2024-03-25T16:36:43.988 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:36:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:44.143 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:44.144 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:44.144 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:44.361 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-mirror 2024-03-25T16:36:44.938 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:45.095 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:45.096 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:45.096 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T16:36:45.096 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 3.1 M 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 13 M 2024-03-25T16:36:45.097 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:50.971 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 538 kB/s | 3.1 MB 00:05 2024-03-25T16:36:50.971 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:50.972 INFO:teuthology.orchestra.run.smithi033.stdout:Total 538 kB/s | 3.1 MB 00:05 2024-03-25T16:36:50.972 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:50.984 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:50.984 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:51.065 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:51.065 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:51.512 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:51.571 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:51.596 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:51.596 INFO:teuthology.orchestra.run.smithi033.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:36:51.596 INFO:teuthology.orchestra.run.smithi033.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:36:51.596 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:36:51.596 INFO:teuthology.orchestra.run.smithi033.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:36:51.596 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:52.651 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:52.651 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:52.651 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:52.917 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-nbd 2024-03-25T16:36:53.504 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:54 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:53.659 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:53.660 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:53.660 INFO:teuthology.orchestra.run.smithi033.stdout: Package Architecture Version Repository Size 2024-03-25T16:36:53.660 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 171 k 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 494 k 2024-03-25T16:36:53.661 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:36:54.657 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 172 kB/s | 171 kB 00:00 2024-03-25T16:36:54.657 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:36:54.658 INFO:teuthology.orchestra.run.smithi033.stdout:Total 172 kB/s | 171 kB 00:00 2024-03-25T16:36:54.658 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:36:54.668 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:36:54.668 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:36:54.731 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:36:54.731 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:36:54.885 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:36:54.999 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:55.425 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:36:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:55.706 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:36:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:36:55.706 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:55.707 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:36:55.924 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:36:56.497 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:36:56.652 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:36:56.652 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:56.652 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T16:36:56.652 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:56.652 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout:Install 2 Packages 2024-03-25T16:36:56.653 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:36:56.654 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 91 M 2024-03-25T16:36:56.654 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 498 M 2024-03-25T16:36:56.654 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T16:38:21.348 INFO:teuthology.orchestra.run.smithi033.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 271 kB/s | 22 MB 01:24 2024-03-25T16:38:22.882 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 232 kB/s | 22 MB 01:38 2024-03-25T16:41:16.100 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 256 kB/s | 68 MB 04:32 2024-03-25T16:41:16.123 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:41:16.124 INFO:teuthology.orchestra.run.smithi022.stdout:Total 341 kB/s | 91 MB 04:32 2024-03-25T16:41:16.124 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T16:41:16.124 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T16:41:16.124 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T16:41:16.711 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T16:41:16.711 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T16:41:19.981 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T16:41:25.213 INFO:teuthology.orchestra.run.smithi033.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 260 kB/s | 68 MB 04:28 2024-03-25T16:41:25.214 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:41:25.215 INFO:teuthology.orchestra.run.smithi033.stdout:Total 345 kB/s | 91 MB 04:28 2024-03-25T16:41:25.215 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T16:41:25.220 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T16:41:25.220 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T16:41:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T16:41:25.829 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T16:41:27.487 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:41:27.560 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:29.246 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T16:41:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:30.794 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:41:31.184 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:41:31.184 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:41:31.184 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T16:41:31.184 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:31.184 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:31.184 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:41:31.184 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T16:41:31.424 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:41:32.000 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:10:27 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T16:41:32.155 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T16:41:32.156 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:41:32.156 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T16:41:32.156 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:41:32.156 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T16:41:32.156 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:41:32.156 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:41:32.156 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T16:41:32.157 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T16:41:32.157 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T16:41:32.157 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T16:41:32.157 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 416 M 2024-03-25T16:41:32.157 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.7 G 2024-03-25T16:41:32.157 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T16:41:36.772 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:41:36.841 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:40.175 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:41:40.175 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:41:40.441 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:41:40.441 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:41:40.441 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T16:41:40.441 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:40.441 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:41:40.441 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:41:40.441 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T16:41:40.670 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:41:41.245 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:10:42 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T16:41:41.399 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 416 M 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.7 G 2024-03-25T16:41:41.400 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T17:03:28.916 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 324 kB/s | 416 MB 21:56 2024-03-25T17:03:28.917 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:03:28.917 INFO:teuthology.orchestra.run.smithi022.stdout:Total 324 kB/s | 416 MB 21:56 2024-03-25T17:03:28.917 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T17:03:28.920 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T17:03:28.920 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T17:03:31.507 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T17:03:31.507 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T17:04:16.200 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T17:04:16.311 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:04:23.262 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:04:23.528 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:04:23.528 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:23.528 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T17:04:23.528 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:23.528 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:23.529 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:04:23.810 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:04:24.385 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:33:20 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:04:24.539 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T17:04:24.540 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:24.541 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 1.4 M 2024-03-25T17:04:24.541 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 6.4 M 2024-03-25T17:04:24.541 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T17:04:27.796 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 449 kB/s | 1.4 MB 00:03 2024-03-25T17:04:27.797 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:04:27.797 INFO:teuthology.orchestra.run.smithi022.stdout:Total 449 kB/s | 1.4 MB 00:03 2024-03-25T17:04:27.797 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T17:04:27.801 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T17:04:27.801 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T17:04:27.847 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T17:04:27.847 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T17:04:28.112 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T17:04:28.222 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:04:28.612 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:04:28.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:04:28.894 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:28.895 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T17:04:28.895 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:28.895 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:28.895 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:04:29.103 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:04:29.677 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:33:25 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:04:29.831 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:04:29.832 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T17:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 176 M 2024-03-25T17:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 753 M 2024-03-25T17:04:29.833 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T17:07:25.436 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 276 kB/s | 416 MB 25:44 2024-03-25T17:07:25.436 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:07:25.437 INFO:teuthology.orchestra.run.smithi033.stdout:Total 276 kB/s | 416 MB 25:44 2024-03-25T17:07:25.437 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T17:07:25.439 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T17:07:25.440 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T17:07:28.027 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T17:07:28.027 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T17:08:12.727 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T17:08:12.832 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:08:19.875 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:08:20.133 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:08:20.133 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:20.133 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T17:08:20.133 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:20.133 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:20.133 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:08:20.404 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:08:20.983 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:37:21 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:08:21.139 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:08:21.139 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T17:08:21.139 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:======================================================================================== 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 1.4 M 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 6.4 M 2024-03-25T17:08:21.140 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T17:08:28.340 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 203 kB/s | 1.4 MB 00:07 2024-03-25T17:08:28.341 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:28.341 INFO:teuthology.orchestra.run.smithi033.stdout:Total 203 kB/s | 1.4 MB 00:07 2024-03-25T17:08:28.341 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T17:08:28.345 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T17:08:28.346 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T17:08:28.392 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T17:08:28.393 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T17:08:28.659 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T17:08:28.765 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:08:29.173 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:08:29.446 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:08:29.446 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:29.446 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T17:08:29.446 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:29.446 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:29.446 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:08:29.612 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:08:30.186 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:37:31 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:08:30.339 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:08:30.340 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:08:30.340 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T17:08:30.340 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 176 M 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 753 M 2024-03-25T17:08:30.341 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T17:15:40.738 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 268 kB/s | 176 MB 11:10 2024-03-25T17:15:40.738 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:40.738 INFO:teuthology.orchestra.run.smithi022.stdout:Total 268 kB/s | 176 MB 11:10 2024-03-25T17:15:40.738 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T17:15:40.741 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T17:15:40.742 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T17:15:41.850 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T17:15:41.851 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T17:16:00.888 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T17:16:00.986 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:16:05.675 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:16:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:16:05.944 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:16:05.945 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T17:16:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:05.945 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:16:05.945 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:16:06.158 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:16:06.737 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:45:02 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:16:06.891 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T17:16:06.892 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:16:06.893 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T17:16:06.893 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:16:06.893 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 672 M 2024-03-25T17:16:06.893 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 3.1 G 2024-03-25T17:16:06.893 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T17:20:14.291 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 256 kB/s | 176 MB 11:43 2024-03-25T17:20:14.308 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:20:14.308 INFO:teuthology.orchestra.run.smithi033.stdout:Total 256 kB/s | 176 MB 11:43 2024-03-25T17:20:14.308 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T17:20:14.308 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T17:20:14.308 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T17:20:15.409 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T17:20:15.409 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T17:20:34.452 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T17:20:34.547 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:20:39.322 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:20:39.583 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:20:39.583 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:20:39.583 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T17:20:39.584 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:39.584 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:20:39.584 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:20:39.770 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:20:40.345 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 0:49:41 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:20:40.500 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:20:40.501 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:20:40.501 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T17:20:40.501 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:20:40.501 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T17:20:40.501 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 672 M 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 3.1 G 2024-03-25T17:20:40.502 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T17:39:37.843 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 488 kB/s | 672 MB 23:30 2024-03-25T17:39:37.873 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:37.873 INFO:teuthology.orchestra.run.smithi022.stdout:Total 488 kB/s | 672 MB 23:30 2024-03-25T17:39:37.873 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T17:39:37.873 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T17:39:37.874 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T17:39:42.007 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T17:39:42.007 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T17:40:57.183 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T17:40:57.737 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:41:06.635 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:41:06.914 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:41:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:06.915 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T17:41:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:06.915 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:06.915 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:41:07.215 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:41:07.793 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:10:03 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:41:07.891 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:41:07.946 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:41:07.947 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-25T17:41:07.947 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:41:08.023 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:41:08.601 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:10:04 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:41:08.756 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 20 M 2024-03-25T17:41:08.757 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 103 M 2024-03-25T17:41:08.758 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T17:41:45.757 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 545 kB/s | 20 MB 00:36 2024-03-25T17:41:45.758 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:45.758 INFO:teuthology.orchestra.run.smithi022.stdout:Total 545 kB/s | 20 MB 00:37 2024-03-25T17:41:45.759 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T17:41:45.762 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T17:41:45.762 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T17:41:45.917 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T17:41:45.917 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T17:41:48.272 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T17:41:48.366 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:49.367 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:49.639 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:49.639 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:49.639 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T17:41:49.639 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:49.639 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:49.639 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:41:49.938 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:41:50.513 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:10:46 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:41:50.668 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:41:50.669 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:41:50.669 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T17:41:50.669 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:41:50.669 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 48 M 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 173 M 2024-03-25T17:41:50.670 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T17:43:40.990 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 499 kB/s | 672 MB 23:00 2024-03-25T17:43:40.991 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:40.991 INFO:teuthology.orchestra.run.smithi033.stdout:Total 499 kB/s | 672 MB 23:00 2024-03-25T17:43:40.991 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T17:43:40.996 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T17:43:40.996 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T17:43:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T17:43:45.150 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T17:43:57.492 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 385 kB/s | 48 MB 02:06 2024-03-25T17:43:57.493 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:57.493 INFO:teuthology.orchestra.run.smithi022.stdout:Total 385 kB/s | 48 MB 02:06 2024-03-25T17:43:57.493 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T17:43:57.497 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T17:43:57.497 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T17:43:57.825 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T17:43:57.826 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T17:44:02.849 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T17:44:02.935 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:04.295 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:04.578 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:04.578 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:44:04.578 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T17:44:04.578 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:04.578 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:44:04.579 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:44:04.765 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:44:05.341 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:13:01 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:44:05.495 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:44:05.496 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:44:05.496 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T17:44:05.496 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:44:05.496 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T17:44:05.496 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:44:05.496 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:44:05.496 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T17:44:05.497 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:44:05.497 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T17:44:05.497 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:44:05.497 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 99 M 2024-03-25T17:44:05.497 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 449 M 2024-03-25T17:44:05.497 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T17:45:00.211 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T17:45:00.341 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:08.036 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:08.298 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:45:08.299 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:08.299 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T17:45:08.299 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:08.299 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:08.299 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:45:08.563 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:45:09.140 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:14:10 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:45:09.239 INFO:teuthology.orchestra.run.smithi033.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:45:09.294 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:45:09.295 INFO:teuthology.orchestra.run.smithi033.stdout:Nothing to do. 2024-03-25T17:45:09.295 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:45:09.363 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:45:09.941 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:14:10 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:10.098 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T17:45:10.099 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:45:10.099 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T17:45:10.099 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:10.099 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 20 M 2024-03-25T17:45:10.099 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 103 M 2024-03-25T17:45:10.099 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T17:45:54.336 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 456 kB/s | 20 MB 00:44 2024-03-25T17:45:54.336 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:54.337 INFO:teuthology.orchestra.run.smithi033.stdout:Total 456 kB/s | 20 MB 00:44 2024-03-25T17:45:54.337 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T17:45:54.340 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T17:45:54.340 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T17:45:54.496 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T17:45:54.496 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T17:45:56.845 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T17:45:56.952 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:57.864 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:58.126 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:58.126 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:58.126 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T17:45:58.126 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:58.126 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:58.126 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:45:58.288 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:45:58.865 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:14:59 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:45:59.020 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:45:59.020 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 48 M 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 173 M 2024-03-25T17:45:59.021 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T17:48:53.214 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 281 kB/s | 48 MB 02:54 2024-03-25T17:48:53.214 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:48:53.214 INFO:teuthology.orchestra.run.smithi033.stdout:Total 281 kB/s | 48 MB 02:54 2024-03-25T17:48:53.214 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T17:48:53.217 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T17:48:53.217 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T17:48:53.544 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T17:48:53.544 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T17:48:58.565 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T17:48:58.664 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:00.015 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:00.282 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:00.282 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:49:00.282 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T17:49:00.282 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:00.283 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:49:00.283 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:49:00.444 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:49:01.022 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:18:02 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:49:01.183 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:49:01.184 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T17:49:01.185 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:49:01.185 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 99 M 2024-03-25T17:49:01.185 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 449 M 2024-03-25T17:49:01.185 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T17:49:48.987 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 294 kB/s | 99 MB 05:43 2024-03-25T17:49:48.988 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:48.988 INFO:teuthology.orchestra.run.smithi022.stdout:Total 294 kB/s | 99 MB 05:43 2024-03-25T17:49:48.988 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T17:49:48.991 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T17:49:48.991 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T17:49:49.628 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T17:49:49.628 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T17:50:00.512 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T17:50:00.620 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:50:03.522 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:50:03.803 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:50:03.803 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:50:03.803 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T17:50:03.803 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:50:03.803 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:50:03.803 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T17:50:04.009 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:50:04.589 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:19:00 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T17:50:04.743 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T17:50:04.744 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:50:04.744 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T17:50:04.745 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T17:50:04.746 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 287 M 2024-03-25T17:50:04.746 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.2 G 2024-03-25T17:50:04.746 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T17:53:34.918 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 369 kB/s | 99 MB 04:33 2024-03-25T17:53:34.919 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:34.919 INFO:teuthology.orchestra.run.smithi033.stdout:Total 369 kB/s | 99 MB 04:33 2024-03-25T17:53:34.919 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T17:53:34.921 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T17:53:34.921 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T17:53:35.559 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T17:53:35.559 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T17:53:46.423 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T17:53:46.525 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:49.432 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:49.813 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:49.813 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:53:49.813 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T17:53:49.813 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:49.814 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:53:49.814 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T17:53:49.994 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:53:50.574 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:22:51 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T17:53:50.727 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T17:53:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:53:50.728 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T17:53:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:53:50.728 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 287 M 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.2 G 2024-03-25T17:53:50.729 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:03:06.162 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 376 kB/s | 287 MB 13:01 2024-03-25T18:03:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:Total 376 kB/s | 287 MB 13:01 2024-03-25T18:03:06.163 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:03:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:03:06.165 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:03:07.954 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:03:07.955 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:03:38.995 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:03:39.107 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:46.548 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:46.837 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:46.837 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:03:46.837 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:03:46.837 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:46.837 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:03:46.837 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:03:47.080 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:03:47.654 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:32:43 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:03:47.810 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:03:47.811 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:03:47.812 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:03:47.812 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 5.9 M 2024-03-25T18:03:47.812 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 28 M 2024-03-25T18:03:47.812 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:04:03.082 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 399 kB/s | 5.9 MB 00:15 2024-03-25T18:04:03.083 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:03.083 INFO:teuthology.orchestra.run.smithi022.stdout:Total 398 kB/s | 5.9 MB 00:15 2024-03-25T18:04:03.083 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:04:03.087 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:04:03.087 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:04:03.162 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:04:03.163 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:04:03.935 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:04:04.041 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:04.610 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:04.897 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:04.897 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:04.897 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:04:04.897 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:04.897 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:04.897 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:04:05.109 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:04:05.686 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:33:01 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:04:05.841 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:04:05.842 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 109 k 2024-03-25T18:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 440 k 2024-03-25T18:04:05.843 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:04:06.535 INFO:teuthology.orchestra.run.smithi022.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 157 kB/s | 109 kB 00:00 2024-03-25T18:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout:Total 157 kB/s | 109 kB 00:00 2024-03-25T18:04:06.536 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:04:06.540 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:04:06.540 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:04:06.577 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:04:06.577 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:04:06.697 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:04:06.804 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:07.175 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:07.455 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:07.455 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:07.455 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:04:07.455 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:07.455 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:07.455 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:04:07.642 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:04:08.218 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:33:04 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:04:08.373 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:04:08.373 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:04:08.374 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:08.375 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 5.4 M 2024-03-25T18:04:08.375 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 25 M 2024-03-25T18:04:08.375 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:04:24.977 INFO:teuthology.orchestra.run.smithi022.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 330 kB/s | 5.4 MB 00:16 2024-03-25T18:04:24.978 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:24.978 INFO:teuthology.orchestra.run.smithi022.stdout:Total 330 kB/s | 5.4 MB 00:16 2024-03-25T18:04:24.979 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:04:24.983 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:04:24.983 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:04:25.054 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:04:25.054 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:04:25.755 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:04:25.870 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:26.367 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:26.652 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:26.652 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:26.652 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:04:26.652 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:26.652 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:26.653 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:04:26.845 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados2-debuginfo 2024-03-25T18:04:27.421 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:33:23 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:04:27.575 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:04:27.576 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 44 M 2024-03-25T18:04:27.577 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 253 M 2024-03-25T18:04:27.577 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:07:11.808 INFO:teuthology.orchestra.run.smithi022.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 276 kB/s | 44 MB 02:44 2024-03-25T18:07:11.809 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:11.809 INFO:teuthology.orchestra.run.smithi022.stdout:Total 276 kB/s | 44 MB 02:44 2024-03-25T18:07:11.809 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:07:11.812 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:07:11.812 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:07:12.125 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:07:17.497 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:07:17.592 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:19.402 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:19.691 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:19.691 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:07:19.692 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:07:19.692 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:19.692 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:07:19.692 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:07:19.898 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:07:20.471 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:36:16 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:07:20.630 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:07:20.630 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:07:20.630 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 52 M 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 308 M 2024-03-25T18:07:20.631 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:10:09.046 INFO:teuthology.orchestra.run.smithi022.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 315 kB/s | 52 MB 02:48 2024-03-25T18:10:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:Total 315 kB/s | 52 MB 02:48 2024-03-25T18:10:09.047 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:10:09.049 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:10:09.049 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:10:09.407 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:10:09.408 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:10:15.752 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:10:15.861 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:17.944 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:18.218 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:10:18.218 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:18.218 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:10:18.218 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:18.219 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:18.219 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:10:18.406 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:10:18.982 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:39:14 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:10:19.137 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:10:19.137 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:19.137 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 615 k 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 2.1 M 2024-03-25T18:10:19.138 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:10:20.905 INFO:teuthology.orchestra.run.smithi022.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 348 kB/s | 615 kB 00:01 2024-03-25T18:10:20.906 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:20.906 INFO:teuthology.orchestra.run.smithi022.stdout:Total 348 kB/s | 615 kB 00:01 2024-03-25T18:10:20.906 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:10:20.910 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:10:20.910 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:10:20.954 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:10:20.954 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:10:21.130 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:10:21.223 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:21.609 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:10:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:21.894 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:21.894 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:10:22.072 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:10:22.646 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:39:18 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:10:22.802 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:10:22.803 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:22.804 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:10:22.804 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:22.804 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 771 k 2024-03-25T18:10:22.804 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 2.7 M 2024-03-25T18:10:22.804 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:10:25.216 INFO:teuthology.orchestra.run.smithi022.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 320 kB/s | 771 kB 00:02 2024-03-25T18:10:25.217 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:25.217 INFO:teuthology.orchestra.run.smithi022.stdout:Total 319 kB/s | 771 kB 00:02 2024-03-25T18:10:25.217 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:10:25.222 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:10:25.222 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:10:25.265 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:10:25.266 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:10:25.454 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:10:25.568 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:25.925 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:26.200 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:10:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:26.200 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:26.200 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:10:26.414 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:10:26.987 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:39:22 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:10:27.142 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:27.143 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 1.1 M 2024-03-25T18:10:27.144 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 4.3 M 2024-03-25T18:10:27.144 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:10:30.702 INFO:teuthology.orchestra.run.smithi022.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 323 kB/s | 1.1 MB 00:03 2024-03-25T18:10:30.702 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:30.703 INFO:teuthology.orchestra.run.smithi022.stdout:Total 323 kB/s | 1.1 MB 00:03 2024-03-25T18:10:30.703 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:10:30.707 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:10:30.707 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:10:30.753 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:10:30.753 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:10:30.986 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:10:31.096 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:31.496 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:31.784 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:31.784 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:31.784 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:10:31.784 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:31.784 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:31.784 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:10:32.021 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:10:32.596 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:39:28 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:10:32.752 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:10:32.753 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:32.754 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 337 k 2024-03-25T18:10:32.754 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.0 M 2024-03-25T18:10:32.754 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:10:34.289 INFO:teuthology.orchestra.run.smithi022.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 220 kB/s | 337 kB 00:01 2024-03-25T18:10:34.290 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:34.290 INFO:teuthology.orchestra.run.smithi022.stdout:Total 219 kB/s | 337 kB 00:01 2024-03-25T18:10:34.290 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:10:34.294 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:10:34.295 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:10:34.337 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:10:34.337 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:10:34.502 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:10:34.608 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:34.979 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:35.267 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:35.267 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:35.267 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:10:35.267 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:35.267 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:35.267 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:10:35.474 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:10:36.047 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:39:32 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:10:36.201 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:10:36.202 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:36.203 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 509 k 2024-03-25T18:10:36.203 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 2.5 M 2024-03-25T18:10:36.203 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:10:37.892 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 301 kB/s | 509 kB 00:01 2024-03-25T18:10:37.892 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:37.893 INFO:teuthology.orchestra.run.smithi022.stdout:Total 301 kB/s | 509 kB 00:01 2024-03-25T18:10:37.893 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:10:37.897 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:10:37.897 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:10:37.939 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:10:37.939 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:10:38.107 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:10:38.211 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:38.583 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:38.866 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:38.867 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:10:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:38.867 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:38.867 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:10:39.067 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:10:39.405 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 291 kB/s | 287 MB 16:48 2024-03-25T18:10:39.405 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:39.406 INFO:teuthology.orchestra.run.smithi033.stdout:Total 291 kB/s | 287 MB 16:48 2024-03-25T18:10:39.406 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:10:39.410 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:10:39.410 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:10:39.641 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:39:35 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:10:39.798 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:10:39.798 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 52 M 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 298 M 2024-03-25T18:10:39.799 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:10:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:10:41.200 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:11:12.311 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:11:12.946 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:11:16.380 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:11:16.667 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:11:16.667 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:16.667 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:11:16.667 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:16.667 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:16.667 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:11:17.009 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:11:17.594 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:40:18 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:11:17.753 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:11:17.755 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:17.755 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:11:17.755 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:17.755 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:11:17.755 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:11:17.755 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:17.756 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:11:17.756 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:17.756 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:11:17.756 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:17.756 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 5.9 M 2024-03-25T18:11:17.756 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 28 M 2024-03-25T18:11:17.757 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:11:39.469 INFO:teuthology.orchestra.run.smithi033.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 280 kB/s | 5.9 MB 00:21 2024-03-25T18:11:39.470 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:39.470 INFO:teuthology.orchestra.run.smithi033.stdout:Total 280 kB/s | 5.9 MB 00:21 2024-03-25T18:11:39.470 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:11:39.474 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:11:39.475 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:11:39.550 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:11:39.550 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:11:40.319 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:11:40.419 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:11:41.068 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:11:41.338 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:11:41.338 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:41.338 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:11:41.338 INFO:teuthology.orchestra.run.smithi033.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:41.338 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:41.338 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:11:41.523 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:11:42.097 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:40:43 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:11:42.253 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:42.254 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 109 k 2024-03-25T18:11:42.255 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 440 k 2024-03-25T18:11:42.255 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:11:43.254 INFO:teuthology.orchestra.run.smithi033.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 109 kB/s | 109 kB 00:00 2024-03-25T18:11:43.255 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:11:43.255 INFO:teuthology.orchestra.run.smithi033.stdout:Total 109 kB/s | 109 kB 00:01 2024-03-25T18:11:43.256 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:11:43.260 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:11:43.260 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:11:43.299 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:11:43.300 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:11:43.420 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:11:43.531 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:43.910 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:44.170 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:11:44.171 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:11:44.171 INFO:teuthology.orchestra.run.smithi033.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:11:44.171 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:44.171 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:11:44.364 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:11:44.938 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:40:45 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:11:45.093 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:11:45.094 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:11:45.095 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 5.4 M 2024-03-25T18:11:45.095 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 25 M 2024-03-25T18:11:45.095 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:12:01.580 INFO:teuthology.orchestra.run.smithi033.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 332 kB/s | 5.4 MB 00:16 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi033.stdout:Total 332 kB/s | 5.4 MB 00:16 2024-03-25T18:12:01.581 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:12:01.586 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:12:01.586 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:12:01.657 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:12:01.657 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:12:02.362 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:12:02.465 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:12:02.994 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:12:03.269 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:12:03.269 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:12:03.269 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:12:03.269 INFO:teuthology.orchestra.run.smithi033.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:03.269 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:12:03.269 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:12:03.463 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librados2-debuginfo 2024-03-25T18:12:04.039 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:41:05 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:12:04.195 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:12:04.195 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 44 M 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 253 M 2024-03-25T18:12:04.196 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:13:03.343 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 368 kB/s | 52 MB 02:23 2024-03-25T18:13:03.344 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:03.344 INFO:teuthology.orchestra.run.smithi022.stdout:Total 368 kB/s | 52 MB 02:23 2024-03-25T18:13:03.344 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:13:03.347 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:13:03.347 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:13:03.696 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:13:03.697 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:13:09.944 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:13:10.035 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:13:12.301 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:13:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:12.301 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:13:12.301 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:13:12.516 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:13:13.094 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 1:42:09 ago on Mon 25 Mar 2024 04:31:04 PM UTC. 2024-03-25T18:13:13.250 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 1.3 M 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 5.5 M 2024-03-25T18:13:13.251 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-25T18:13:18.469 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 246 kB/s | 1.3 MB 00:05 2024-03-25T18:13:18.470 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:18.470 INFO:teuthology.orchestra.run.smithi022.stdout:Total 246 kB/s | 1.3 MB 00:05 2024-03-25T18:13:18.470 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-25T18:13:18.475 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-25T18:13:18.475 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-25T18:13:18.522 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-25T18:13:18.522 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-25T18:13:18.772 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-25T18:13:18.872 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:19.301 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:19.590 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:19.590 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:13:19.591 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-25T18:13:19.591 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:19.591 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:13:19.591 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-25T18:13:19.797 DEBUG:teuthology.parallel:result is None 2024-03-25T18:13:57.933 INFO:teuthology.orchestra.run.smithi033.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 399 kB/s | 44 MB 01:53 2024-03-25T18:13:57.934 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:57.935 INFO:teuthology.orchestra.run.smithi033.stdout:Total 399 kB/s | 44 MB 01:53 2024-03-25T18:13:57.935 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:13:57.938 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:13:57.938 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:13:58.250 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:13:58.250 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:14:03.585 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:14:03.692 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:14:05.542 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:14:05.809 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:14:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:14:05.810 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:14:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:14:05.810 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:14:05.810 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:14:06.014 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:14:06.589 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:43:07 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:14:06.754 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:14:06.755 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:14:06.756 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:14:06.756 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 52 M 2024-03-25T18:14:06.756 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 308 M 2024-03-25T18:14:06.756 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:15:33.076 INFO:teuthology.orchestra.run.smithi033.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 615 kB/s | 52 MB 01:26 2024-03-25T18:15:33.077 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:33.077 INFO:teuthology.orchestra.run.smithi033.stdout:Total 615 kB/s | 52 MB 01:26 2024-03-25T18:15:33.077 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:15:33.080 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:15:33.080 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:15:33.439 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:15:33.439 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:15:39.787 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:15:39.902 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:42.055 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:42.324 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:42.324 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:42.324 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:15:42.324 INFO:teuthology.orchestra.run.smithi033.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:42.324 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:42.324 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:15:42.541 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:15:43.110 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:44:44 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:15:43.266 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:15:43.267 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:43.267 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:15:43.267 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:43.268 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:15:43.268 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:15:43.268 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:43.268 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:15:43.268 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:43.268 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:15:43.269 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:43.269 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 615 k 2024-03-25T18:15:43.269 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 2.1 M 2024-03-25T18:15:43.269 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:15:46.703 INFO:teuthology.orchestra.run.smithi033.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 179 kB/s | 615 kB 00:03 2024-03-25T18:15:46.703 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:46.703 INFO:teuthology.orchestra.run.smithi033.stdout:Total 179 kB/s | 615 kB 00:03 2024-03-25T18:15:46.704 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:15:46.708 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:15:46.708 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:15:46.751 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:15:46.752 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:15:46.925 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:15:47.038 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:47.407 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:47.693 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:15:47.693 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:47.693 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:15:47.693 INFO:teuthology.orchestra.run.smithi033.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:47.693 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:47.693 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:15:47.887 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:15:48.464 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:44:49 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:15:48.620 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:15:48.620 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:48.620 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:15:48.620 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:48.620 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 771 k 2024-03-25T18:15:48.621 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 2.7 M 2024-03-25T18:15:48.622 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:15:50.389 INFO:teuthology.orchestra.run.smithi033.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 436 kB/s | 771 kB 00:01 2024-03-25T18:15:50.390 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:50.391 INFO:teuthology.orchestra.run.smithi033.stdout:Total 436 kB/s | 771 kB 00:01 2024-03-25T18:15:50.391 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:15:50.395 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:15:50.395 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:15:50.439 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:15:50.439 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:15:50.632 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:15:50.741 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:51.116 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:51.381 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:15:51.381 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:51.381 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:15:51.382 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:51.382 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:51.382 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:15:51.582 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:15:52.157 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:44:53 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:15:52.314 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:15:52.314 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 1.1 M 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 4.3 M 2024-03-25T18:15:52.315 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:15:54.468 INFO:teuthology.orchestra.run.smithi033.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 535 kB/s | 1.1 MB 00:02 2024-03-25T18:15:54.469 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:54.469 INFO:teuthology.orchestra.run.smithi033.stdout:Total 534 kB/s | 1.1 MB 00:02 2024-03-25T18:15:54.469 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:15:54.473 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:15:54.473 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:15:54.518 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:15:54.518 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:15:54.747 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:15:54.836 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:55.238 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:55.573 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:15:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:55.573 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:55.573 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:15:55.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:15:56.519 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:44:57 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:15:56.674 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:15:56.675 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 337 k 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 1.0 M 2024-03-25T18:15:56.676 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:15:58.437 INFO:teuthology.orchestra.run.smithi033.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 192 kB/s | 337 kB 00:01 2024-03-25T18:15:58.438 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:58.438 INFO:teuthology.orchestra.run.smithi033.stdout:Total 191 kB/s | 337 kB 00:01 2024-03-25T18:15:58.439 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:15:58.442 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:15:58.443 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:15:58.485 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:15:58.486 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:15:58.629 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:15:58.739 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:59.116 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:15:59.388 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:59.389 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:15:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:59.389 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:15:59.389 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:15:59.614 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:16:00.191 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:45:01 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:16:00.346 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:16:00.347 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:16:00.347 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:16:00.347 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:16:00.347 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:16:00.347 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 509 k 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 2.5 M 2024-03-25T18:16:00.348 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:16:02.357 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 253 kB/s | 509 kB 00:02 2024-03-25T18:16:02.358 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:02.358 INFO:teuthology.orchestra.run.smithi033.stdout:Total 253 kB/s | 509 kB 00:02 2024-03-25T18:16:02.359 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:16:02.363 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:16:02.363 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:16:02.405 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:16:02.405 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:16:02.578 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:16:02.677 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:03.039 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:03.302 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:03.302 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:16:03.303 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:16:03.303 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:03.303 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:16:03.303 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:16:03.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:16:04.068 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:45:05 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:16:04.224 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:16:04.225 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:16:04.226 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 52 M 2024-03-25T18:16:04.226 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 298 M 2024-03-25T18:16:04.226 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:18:10.019 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 420 kB/s | 52 MB 02:05 2024-03-25T18:18:10.020 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:10.020 INFO:teuthology.orchestra.run.smithi033.stdout:Total 420 kB/s | 52 MB 02:05 2024-03-25T18:18:10.020 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:18:10.023 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:18:10.023 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:18:10.374 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:18:10.374 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:18:16.625 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:18:16.735 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:18.779 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:19.093 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:18:19.093 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:19.093 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:18:19.093 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:19.093 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:19.093 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:18:19.282 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:18:19.862 INFO:teuthology.orchestra.run.smithi033.stdout:Last metadata expiration check: 1:47:20 ago on Mon 25 Mar 2024 04:30:59 PM UTC. 2024-03-25T18:18:20.018 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-03-25T18:18:20.018 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:18:20.018 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repo Size 2024-03-25T18:18:20.018 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:18:20.018 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-03-25T18:18:20.018 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout:Install 1 Package 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 1.3 M 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 5.5 M 2024-03-25T18:18:20.019 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-03-25T18:18:21.911 INFO:teuthology.orchestra.run.smithi033.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 678 kB/s | 1.3 MB 00:01 2024-03-25T18:18:21.912 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:21.912 INFO:teuthology.orchestra.run.smithi033.stdout:Total 677 kB/s | 1.3 MB 00:01 2024-03-25T18:18:21.913 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-03-25T18:18:21.917 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-03-25T18:18:21.917 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-03-25T18:18:21.963 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-03-25T18:18:21.964 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-03-25T18:18:22.219 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-03-25T18:18:22.321 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:22.749 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:23.023 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:18:23.023 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:23.023 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-03-25T18:18:23.023 INFO:teuthology.orchestra.run.smithi033.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:23.024 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:23.024 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-03-25T18:18:23.232 DEBUG:teuthology.parallel:result is None 2024-03-25T18:18:23.233 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:18:23.373 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:18:23.396 INFO:teuthology.orchestra.run.smithi022.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:23.397 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:23.397 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:18:23.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:18:23.537 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:18:23.557 INFO:teuthology.orchestra.run.smithi033.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:23.557 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:18:23.557 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:18:23.559 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:18:23.559 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:23.559 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:18:23.590 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:23.591 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:18:23.625 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:18:23.625 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:23.625 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:18:23.662 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:18:23.733 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:23.733 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:18:23.760 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:18:23.826 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:18:23.826 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:23.826 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:18:23.856 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:18:23.927 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:23.927 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:18:23.952 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:18:24.019 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:18:24.019 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:24.019 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:18:24.049 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:18:24.118 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:24.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:18:24.144 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:18:24.210 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:18:24.288 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:18:24.288 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /var/log/ceph 2024-03-25T18:18:24.291 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /var/log/ceph 2024-03-25T18:18:24.318 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:18:24.318 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:18:24.360 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:18:24.387 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:18:24.387 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:18:24.428 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:18:24.481 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:18:24.481 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:18:24.481 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621275', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi022.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOqIqfEQZt615P3doFwq9RpYUjamDcNMxl4wqu0jpH8yYxlrfRlZ9eQhZSGpvnHoa8mNuD8n7zg2fH01SeUZWE=', 'smithi033.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCi6NsqnuXO4iVWJhs8L6iaW9zhmGjeZxY8mKhUJZvmHnPsroUzqgUn1tmOoUIKd0cSSaEEofFq8BfyG5kZ8ksI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:18:24.481 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:18:24.496 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:18:24.510 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:18:24.551 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:18:24.577 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:24.577 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:18:24.633 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:18:24.633 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-25 16:36:39.080474988 +0000 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-25 16:29:55.391441317 +0000 2024-03-25T18:18:24.688 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-25 16:29:55.391441317 +0000 2024-03-25T18:18:24.689 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-25 16:29:55.391441317 +0000 2024-03-25T18:18:24.689 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:18:24.754 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-25T18:18:24.754 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-25T18:18:24.754 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000101364 s, 5.1 MB/s 2024-03-25T18:18:24.755 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:18:24.812 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2024-03-25T18:18:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:18:24.867 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:24.867 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:18:24.867 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:24.867 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:24.867 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-25 16:36:39.080474988 +0000 2024-03-25T18:18:24.867 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-25 16:29:55.730436877 +0000 2024-03-25T18:18:24.868 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-25 16:29:55.730436877 +0000 2024-03-25T18:18:24.868 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-25 16:29:55.730436877 +0000 2024-03-25T18:18:24.868 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:18:24.933 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-25T18:18:24.933 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-25T18:18:24.933 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.9674e-05 s, 5.1 MB/s 2024-03-25T18:18:24.934 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:18:24.991 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2024-03-25T18:18:25.045 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:18:25.045 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:25.046 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:18:25.046 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:25.046 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:25.046 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-25 16:36:39.081474975 +0000 2024-03-25T18:18:25.046 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-25 16:29:56.055432621 +0000 2024-03-25T18:18:25.046 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-25 16:29:56.055432621 +0000 2024-03-25T18:18:25.046 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-25 16:29:56.055432621 +0000 2024-03-25T18:18:25.046 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:18:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-25T18:18:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-25T18:18:25.112 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.0222e-05 s, 5.7 MB/s 2024-03-25T18:18:25.113 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:18:25.169 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2024-03-25T18:18:25.223 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:18:25.223 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:25.224 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:18:25.224 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:25.224 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:25.224 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-25 16:36:39.081474975 +0000 2024-03-25T18:18:25.224 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-25 16:29:56.386428286 +0000 2024-03-25T18:18:25.224 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-25 16:29:56.386428286 +0000 2024-03-25T18:18:25.224 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-25 16:29:56.386428286 +0000 2024-03-25T18:18:25.224 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:18:25.289 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-25T18:18:25.289 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-25T18:18:25.289 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.8717e-05 s, 5.2 MB/s 2024-03-25T18:18:25.291 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:18:25.346 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:25.346 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:25.346 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:18:25.364 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:18:25.364 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-03-25T18:18:25.422 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:18:25.422 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:25.422 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:18:25.423 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:25.423 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:25.423 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-25 16:36:51.836041184 +0000 2024-03-25T18:18:25.423 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-25 16:29:55.748015701 +0000 2024-03-25T18:18:25.423 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-25 16:29:55.748015701 +0000 2024-03-25T18:18:25.423 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-25 16:29:55.748015701 +0000 2024-03-25T18:18:25.423 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:18:25.492 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-25T18:18:25.492 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-25T18:18:25.492 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.00051239 s, 999 kB/s 2024-03-25T18:18:25.494 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:18:25.553 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-03-25T18:18:25.610 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:18:25.610 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:25.610 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:18:25.610 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:25.610 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:25.611 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-25 16:36:51.836041184 +0000 2024-03-25T18:18:25.611 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-25 16:29:56.102009791 +0000 2024-03-25T18:18:25.611 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-25 16:29:56.102009791 +0000 2024-03-25T18:18:25.611 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-25 16:29:56.102009791 +0000 2024-03-25T18:18:25.611 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:18:25.680 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-25T18:18:25.680 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-25T18:18:25.680 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000215652 s, 2.4 MB/s 2024-03-25T18:18:25.681 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:18:25.740 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-03-25T18:18:25.797 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:18:25.797 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:25.797 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:18:25.798 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:25.798 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:25.798 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-25 16:36:51.836041184 +0000 2024-03-25T18:18:25.798 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-25 16:29:56.445004064 +0000 2024-03-25T18:18:25.798 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-25 16:29:56.445004064 +0000 2024-03-25T18:18:25.798 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-25 16:29:56.445004064 +0000 2024-03-25T18:18:25.798 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:18:25.867 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-25T18:18:25.867 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-25T18:18:25.867 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000226906 s, 2.3 MB/s 2024-03-25T18:18:25.869 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:18:25.927 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-03-25 16:36:51.837041168 +0000 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-03-25 16:29:56.773998571 +0000 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-03-25 16:29:56.773998571 +0000 2024-03-25T18:18:25.984 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-03-25 16:29:56.773998571 +0000 2024-03-25T18:18:25.985 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:18:26.053 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-03-25T18:18:26.053 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-03-25T18:18:26.053 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000191157 s, 2.7 MB/s 2024-03-25T18:18:26.055 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:18:26.113 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:26.113 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:18:26.113 INFO:tasks.ceph:Generating config... 2024-03-25T18:18:26.114 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:18:26.115 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:18:26.115 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:18:26.115 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:18:26.115 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:18:26.115 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:18:26.115 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:18:26.115 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:18:26.115 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:18:26.115 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:18:26.115 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:18:26.115 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:18:26.115 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:18:26.115 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:18:26.115 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:18:26.115 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:18:26.115 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:18:26.116 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:18:26.116 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:18:26.116 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:18:26.116 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:18:26.116 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:18:26.116 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:18:26.116 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:18:26.117 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:18:26.118 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:18:26.118 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:18:26.118 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:18:26.118 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:18:26.118 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:18:26.180 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:18:26.183 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:18:26.230 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:18:26.257 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.22:6789]'), ('mon.c', '[v1:172.21.15.22:6790]'), ('mon.b', '[v1:172.21.15.33:6789]')] 2024-03-25T18:18:26.257 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.33:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:18:26.258 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:26.258 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:18:26.312 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.22:6789]' --addv c '[v1:172.21.15.22:6790]' --addv b '[v1:172.21.15.33:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:26.397 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:26.397 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: generated fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:18:26.397 INFO:teuthology.orchestra.run.smithi022.stdout:setting min_mon_release = quincy 2024-03-25T18:18:26.397 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 0 2024-03-25T18:18:26.397 INFO:teuthology.orchestra.run.smithi022.stdout:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:min_mon_release 17 (quincy) 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:election_strategy: 3 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:18:26.398 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:18:26.399 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:18:26.413 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 3e36bd18-e4e0-4ff2-83ef-b1243b472434... 2024-03-25T18:18:26.414 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:18:26.455 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:18:26.502 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: chdir = "" 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: auth supported = cephx 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: filestore xattr use omap = true 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: mon clock drift allowed = 1.000 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: osd crush chooseleaf type = 0 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: auth debug = true 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on old message = true 2024-03-25T18:18:26.503 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on bug = true 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: mon pg warn max object skew = 0 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default size = 2 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow primary affinity = true 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow pg remap = true 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:18:26.504 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on no sortbitwise = false 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on too few osds = false 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon_allow_pool_size_one = true 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: osd default data pool replay window = 5 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon allow pool delete = true 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon cluster log file level = debug 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: debug asserts on shutdown = true 2024-03-25T18:18:26.505 INFO:teuthology.orchestra.run.smithi022.stdout: mon health detail to clog = false 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: mon host = "[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.33:6789]" 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: mon client directed command retry = 5 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: mon election default strategy = 3 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: ms bind msgr2 = False 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject socket failures = 5000 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: ms type = async 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: osd heartbeat grace = 80 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout:[osd] 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: osd journal size = 100 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub load threshold = 5.0 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub max interval = 600 2024-03-25T18:18:26.506 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd recover clone overlap = true 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd recovery max chunk = 1048576 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug shutdown = true 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug op order = true 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify stray on activate = true 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug trim objects = true 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd open classes on start = true 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug pg log writeout = true 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: osd map max advance = 10 2024-03-25T18:18:26.507 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: journal zero on create = true 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: filestore apply finisher threads = 3 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: bdev debug aio = true 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug misdirected ops = true 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: bdev async discard = True 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: bdev enable discard = True 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore allocator = stupid 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore block size = 96636764160 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore fsck on mount = True 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore zero block detection = True 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluefs = 1 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluestore = 1 2024-03-25T18:18:26.508 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 20 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: debug osd = 20 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: debug rocksdb = 10 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd full ratio = 0.9 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd class default list = * 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd class load list = * 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd client watch timeout = 120 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify cached snaps = True 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify missing on start = True 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd fast shutdown = False 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd max markdown count = 1000 2024-03-25T18:18:26.509 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock override recovery settings = True 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: osd objectstore = bluestore 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue = debug_random 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue cut off = debug_random 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: osd shutdown pgref assert = True 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr] 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: debug mgr = 20 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: mgr/telemetry/nag = false 2024-03-25T18:18:26.510 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout:[mon] 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: debug paxos = 20 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: mon data avail warn = 5 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: mon mgr mkfs grace = 240 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd reporter subtree level = osd 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd prime pg temp = true 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:18:26.511 INFO:teuthology.orchestra.run.smithi022.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: # 1m isn't quite enough 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon min osdmap epochs = 50 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd crush smoke test = False 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune interval = 2 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune min = 15 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon scrub interval = 300 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on pool no app = False 2024-03-25T18:18:26.512 INFO:teuthology.orchestra.run.smithi022.stdout: paxos service trim min = 10 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout:[client] 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: rgw cache enabled = true 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable ops log = true 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable usage log = true 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 20 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: debug objecter = 20 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout: debug rados = 20 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.a] 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.c] 2024-03-25T18:18:26.513 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.b] 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout:[global] 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: chdir = "" 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: auth supported = cephx 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: filestore xattr use omap = true 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: mon clock drift allowed = 1.000 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.514 INFO:teuthology.orchestra.run.smithi033.stdout: osd crush chooseleaf type = 0 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: auth debug = true 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on old message = true 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: ms die on bug = true 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: mon pg warn max object skew = 0 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default size = 2 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow primary affinity = true 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd allow pg remap = true 2024-03-25T18:18:26.515 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on no sortbitwise = false 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on too few osds = false 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon_allow_pool_size_one = true 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: osd default data pool replay window = 5 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon allow pool delete = true 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: mon cluster log file level = debug 2024-03-25T18:18:26.516 INFO:teuthology.orchestra.run.smithi033.stdout: debug asserts on shutdown = true 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: mon health detail to clog = false 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: mon host = "[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.33:6789]" 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: mon client directed command retry = 5 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: mon election default strategy = 3 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: ms bind msgr2 = False 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: ms inject socket failures = 5000 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: ms type = async 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: osd heartbeat grace = 80 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: fsid = 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout:[osd] 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: osd journal size = 100 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub load threshold = 5.0 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: osd scrub max interval = 600 2024-03-25T18:18:26.517 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd recover clone overlap = true 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd recovery max chunk = 1048576 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug shutdown = true 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug op order = true 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify stray on activate = true 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug trim objects = true 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd open classes on start = true 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug pg log writeout = true 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.518 INFO:teuthology.orchestra.run.smithi033.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: osd map max advance = 10 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: journal zero on create = true 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: filestore apply finisher threads = 3 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: bdev debug aio = true 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug misdirected ops = true 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: bdev async discard = True 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: bdev enable discard = True 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore allocator = stupid 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore block size = 96636764160 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore fsck on mount = True 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: bluestore zero block detection = True 2024-03-25T18:18:26.519 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluefs = 1 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: debug bluestore = 1 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: debug osd = 20 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: debug rocksdb = 10 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd full ratio = 0.9 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: osd class default list = * 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: osd class load list = * 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: osd client watch timeout = 120 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify cached snaps = True 2024-03-25T18:18:26.520 INFO:teuthology.orchestra.run.smithi033.stdout: osd debug verify missing on start = True 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd fast shutdown = False 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd max markdown count = 1000 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd mclock override recovery settings = True 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd objectstore = bluestore 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue = debug_random 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd op queue cut off = debug_random 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: osd shutdown pgref assert = True 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr] 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: debug mgr = 20 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:26.521 INFO:teuthology.orchestra.run.smithi033.stdout: mgr/telemetry/nag = false 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout:[mon] 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: debug mon = 20 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: debug paxos = 20 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: debug auth = 20 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: mon data avail warn = 5 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: mon mgr mkfs grace = 240 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd reporter subtree level = osd 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd prime pg temp = true 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:18:26.522 INFO:teuthology.orchestra.run.smithi033.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: # 1m isn't quite enough 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon min osdmap epochs = 50 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon osd crush smoke test = False 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon osdmap full prune interval = 2 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon osdmap full prune min = 15 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon scrub interval = 300 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: mon warn on pool no app = False 2024-03-25T18:18:26.523 INFO:teuthology.orchestra.run.smithi033.stdout: paxos service trim min = 10 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout:[client] 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: rgw cache enabled = true 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable ops log = true 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: rgw enable usage log = true 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: debug monc = 20 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: debug ms = 1 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: debug objecter = 20 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout: debug rados = 20 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.a] 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.c] 2024-03-25T18:18:26.524 INFO:teuthology.orchestra.run.smithi033.stdout:[mon.b] 2024-03-25T18:18:26.527 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:18:26.527 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:18:26.581 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:18:26.581 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:26.581 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:18:26.596 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:26.596 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:18:26.650 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:18:26.650 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:26.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:18:26.650 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:18:26.730 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:26.730 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:26.784 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:18:26.784 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:26.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:18:26.784 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:18:26.830 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:26.830 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:26.886 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:18:26.887 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:18:26.887 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:18:26.942 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:18:26.944 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:18:27.010 INFO:teuthology.orchestra.run.smithi033.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:18:27.013 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:18:27.013 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:18:27.013 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:18:27.054 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:18:27.070 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:18:27.116 INFO:teuthology.orchestra.run.smithi033.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:18:27.132 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:18:27.132 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi033.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:18:27.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:27.158 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:27.158 INFO:tasks.ceph:role: osd.0 2024-03-25T18:18:27.158 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:18:27.159 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:27.225 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:27.257 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-25T18:18:27.259 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-25T18:18:27.259 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:27.331 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:27.402 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:27.469 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:27.469 INFO:tasks.ceph:role: osd.1 2024-03-25T18:18:27.469 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:18:27.469 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:18:27.535 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:27.536 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:27.567 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-25T18:18:27.569 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-25T18:18:27.569 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:27.644 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:27.714 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:27.780 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:27.780 INFO:tasks.ceph:role: osd.2 2024-03-25T18:18:27.780 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:18:27.780 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:18:27.845 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:27.846 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:27.880 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-25T18:18:27.881 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-25T18:18:27.881 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:27.951 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:28.022 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:28.088 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:28.088 INFO:tasks.ceph:role: osd.3 2024-03-25T18:18:28.089 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:18:28.089 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:28.156 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:28.157 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:28.157 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:28.189 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-25T18:18:28.191 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-25T18:18:28.191 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:28.261 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:28.331 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:28.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:28.427+0000 7efcbe2b3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:18:28.428 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:28.427+0000 7efcbe2b3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:18:28.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:28.427+0000 7efcbe2b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:18:28.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:28.427+0000 7efcbe2b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:18:28.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:28.427+0000 7efcbe2b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:18:28.429 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:28.427+0000 7efcbe2b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:18:31.273 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:18:31.304 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:31.394+0000 7f3f1c410640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:18:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:31.395+0000 7f3f1c410640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:18:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:31.395+0000 7f3f1c410640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:18:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:31.395+0000 7f3f1c410640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:18:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:31.395+0000 7f3f1c410640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:18:31.396 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:31.395+0000 7f3f1c410640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:18:34.244 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:18:34.275 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:34.366 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:34.365+0000 7f9977212640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:18:34.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:34.365+0000 7f9977212640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:18:34.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:34.365+0000 7f9977212640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:18:34.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:34.365+0000 7f9977212640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:18:34.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:34.365+0000 7f9977212640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:18:34.367 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:34.365+0000 7f9977212640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:18:37.213 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:18:37.244 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:37.335 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:37.334+0000 7f4d34c13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:18:37.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:37.334+0000 7f4d34c13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:18:37.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:37.334+0000 7f4d34c13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:18:37.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:37.334+0000 7f4d34c13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:18:37.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:37.334+0000 7f4d34c13640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:18:37.336 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:37.334+0000 7f4d34c13640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:18:40.174 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:18:40.204 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:18:40.235 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:40.236 INFO:tasks.ceph:role: osd.4 2024-03-25T18:18:40.236 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:18:40.236 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:40.314 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:40.827 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-25T18:18:40.829 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-25T18:18:40.829 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:18:40.953 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:18:40.987 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:18:41.056 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:41.057 INFO:tasks.ceph:role: osd.5 2024-03-25T18:18:41.057 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:18:41.057 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:41.133 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:41.134 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:41.589 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-25T18:18:41.592 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-25T18:18:41.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:18:41.692 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:18:41.726 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:18:41.795 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:41.795 INFO:tasks.ceph:role: osd.6 2024-03-25T18:18:41.795 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:18:41.795 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:18:41.871 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:41.871 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:41.871 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:41.871 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:41.871 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:41.872 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:41.872 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:41.872 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:41.872 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:41.872 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:42.361 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-25T18:18:42.363 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-25T18:18:42.363 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:18:42.497 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:18:42.534 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:18:42.604 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:18:42.605 INFO:tasks.ceph:role: osd.7 2024-03-25T18:18:42.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:18:42.605 DEBUG:teuthology.orchestra.run.smithi033:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:18:42.682 INFO:teuthology.orchestra.run.smithi033.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:18:43.174 INFO:teuthology.orchestra.run.smithi033.stdout:Discarding blocks...Done. 2024-03-25T18:18:43.176 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi033.front.sepia.ceph.com -o noatime 2024-03-25T18:18:43.177 DEBUG:teuthology.orchestra.run.smithi033:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:18:43.307 DEBUG:teuthology.orchestra.run.smithi033:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:18:43.344 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:43.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:43.438+0000 7fcaece0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:18:43.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:43.438+0000 7fcaece0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:18:43.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:43.438+0000 7fcaece0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:18:43.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:43.439+0000 7fcaece0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:18:43.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:43.439+0000 7fcaece0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:18:43.440 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:43.439+0000 7fcaece0d640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:18:46.298 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:18:46.329 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:46.420+0000 7fa7cb215640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:18:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:46.421+0000 7fa7cb215640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:18:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:46.421+0000 7fa7cb215640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:18:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:46.421+0000 7fa7cb215640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:18:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:46.421+0000 7fa7cb215640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:18:46.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:46.421+0000 7fa7cb215640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:18:49.271 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:18:49.302 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:49.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:49.393+0000 7feeb7015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:18:49.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:49.394+0000 7feeb7015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:18:49.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:49.394+0000 7feeb7015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:18:49.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:49.394+0000 7feeb7015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:18:49.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:49.394+0000 7feeb7015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:18:49.394 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:49.394+0000 7feeb7015640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:18:52.246 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:18:52.276 DEBUG:teuthology.orchestra.run.smithi033:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:52.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:52.367+0000 7f7520c1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:18:52.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:52.367+0000 7f7520c1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:18:52.367 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:52.367+0000 7f7520c1f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:18:52.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:52.367+0000 7f7520c1f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:18:52.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:52.367+0000 7f7520c1f640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:18:52.368 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:18:52.368+0000 7f7520c1f640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:18:55.215 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:18:55.246 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:18:55.246 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:55.246 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:18:55.278 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:55.278 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:18:55.349 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:55.349 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:18:55.419 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:55.419 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:18:55.488 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:55.488 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:18:55.557 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:55.557 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:18:55.588 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:55.588 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:18:55.658 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:55.658 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:18:55.728 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:55.728 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:18:55.798 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:55.798 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:18:55.866 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:18:55.866 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:18:55.884 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:18:55.884 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:18:55.921 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:18:55.921 DEBUG:teuthology.orchestra.run.smithi022:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:18:55.926 DEBUG:teuthology.orchestra.run.smithi033:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.y] 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDyvwFmz2wtOBAAhiSG1DQZ1sFlXUIGZyO2qg== 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.0] 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQD0vwFmwtGMGRAA/Jl33f/vsUmdNf+qnYDmvg== 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.1] 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQD3vwFm1bucFxAAMLILP9RS1QsFjcwCieTuJw== 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.2] 2024-03-25T18:18:55.965 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQD6vwFmaDraFRAA8Iq4Ju8UeMvePsTATgsrwg== 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.3] 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQD9vwFm9NUBFBAAmEqcV/d59bvcYwKzDGWeFg== 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.x] 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDzvwFmGv+hABAA08ua4xIy7CTk1ZEMlAe+Qg== 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.4] 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQADwAFmiJg3GhAANQ1MLlafcnFAjCM1DeUM7w== 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.5] 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAGwAFmdkUYGRAAwQXH0qg+FbkIlGYKbmGT9A== 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.6] 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAJwAFmE8l8FxAAo+MGkDaNcQRgdKw1At40cg== 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.7] 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQAMwAFmIg3qFRAA4Z0ru1zQhtP15wWNO2j2Qg== 2024-03-25T18:18:55.966 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2024-03-25T18:18:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDzvwFm3pxMAxAAiLAjCuKd8C0+AEwisXiodw== 2024-03-25T18:18:55.967 INFO:teuthology.orchestra.run.smithi022.stdout:[client.1] 2024-03-25T18:18:55.967 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDzvwFmAxb0BhAAYZbeJKT01pSh3UPFt5/u2w== 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.y] 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDyvwFmz2wtOBAAhiSG1DQZ1sFlXUIGZyO2qg== 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.0] 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQD0vwFmwtGMGRAA/Jl33f/vsUmdNf+qnYDmvg== 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.1] 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQD3vwFm1bucFxAAMLILP9RS1QsFjcwCieTuJw== 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.2] 2024-03-25T18:18:55.988 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQD6vwFmaDraFRAA8Iq4Ju8UeMvePsTATgsrwg== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.3] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQD9vwFm9NUBFBAAmEqcV/d59bvcYwKzDGWeFg== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[mgr.x] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDzvwFmGv+hABAA08ua4xIy7CTk1ZEMlAe+Qg== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.4] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQADwAFmiJg3GhAANQ1MLlafcnFAjCM1DeUM7w== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.5] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAGwAFmdkUYGRAAwQXH0qg+FbkIlGYKbmGT9A== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.6] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAJwAFmE8l8FxAAo+MGkDaNcQRgdKw1At40cg== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[osd.7] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQAMwAFmIg3qFRAA4Z0ru1zQhtP15wWNO2j2Qg== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[client.0] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDzvwFm3pxMAxAAiLAjCuKd8C0+AEwisXiodw== 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout:[client.1] 2024-03-25T18:18:55.989 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQDzvwFmAxb0BhAAYZbeJKT01pSh3UPFt5/u2w== 2024-03-25T18:18:55.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:18:56.010 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:18:56.086 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.088 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.140 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.142 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.193 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.196 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.247 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.249 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.300 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:18:56.302 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:18:56.353 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.355 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.406 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.408 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.459 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.462 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.513 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.515 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:18:56.565 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:18:56.568 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:18:56.622 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:18:56.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:18:56.679 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:18:56.679 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:18:56.705 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:18:56.802 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:18:56.828 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:18:56.894 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:18:56.990 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:18:57.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:18:57.043 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:18:57.141 DEBUG:teuthology.orchestra.run.smithi033:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:18:57.168 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:57.170 DEBUG:teuthology.orchestra.run.smithi033:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:18:57.222 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:18:57.223 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:18:57.223 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:18:57.223 DEBUG:teuthology.orchestra.run.smithi022:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:18:57.225 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:18:57.225 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:18:57.226 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:18:57.226 DEBUG:teuthology.orchestra.run.smithi022:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:18:57.228 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:18:57.228 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:18:57.228 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:18:57.228 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:18:57.265 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:18:57.265 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:18:57.265 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:18:57.266 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:18:57.268 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:18:57.268 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:18:57.268 DEBUG:teuthology.orchestra.run.smithi033:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:18:57.270 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:18:57.271 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:18:57.271 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:18:57.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.372+0000 7f238c7ba640 10 monclient: get_monmap_and_config 2024-03-25T18:18:57.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.372+0000 7f238c7ba640 10 monclient: build_initial_monmap 2024-03-25T18:18:57.372 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.372+0000 7f238c7ba640 10 monclient: monmap: 2024-03-25T18:18:57.372 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:57.372916+0000 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:57.372916+0000 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:18:57.373 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.373+0000 7f238c7ba640 1 Processor -- start 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.373+0000 7f238c7ba640 1 -- start start 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.373+0000 7f238c7ba640 10 monclient: init 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.373+0000 7f238c7ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.373+0000 7f238c7ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 10 monclient(hunting): picked mon.noname-b con 0x7f23840a9920 addr v1:172.21.15.22:6790/0 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 10 monclient(hunting): picked mon.noname-c con 0x7f23840b45a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 10 monclient(hunting): picked mon.noname-a con 0x7f2384146450 addr v1:172.21.15.22:6789/0 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238406b100 con 0x7f2384146450 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23840b2940 con 0x7f23840b45a0 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238413fec0 con 0x7f23840a9920 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238c7ba640 10 monclient(hunting): authenticate will time out at 7136.596688s 2024-03-25T18:18:57.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f238414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:18:57.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f238413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:18:57.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.374+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f2384145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:18:57.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.575+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f238414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:18:57.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.575+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f238413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:18:57.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.575+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f2384145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:18:57.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.976+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f238414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:18:57.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.976+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f238413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:18:57.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:57.976+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f2384145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:18:58.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:58.777+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f238413f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:18:58.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:58.777+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f238414dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:18:58.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:18:58.777+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f2384145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:00.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient(hunting): tick 2024-03-25T18:19:00.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:00.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f238413f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f2384145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f238414dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f23840b45a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2384146450 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f23840a9920 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238406b100 con 0x7f2384146450 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23840b2940 con 0x7f23840b45a0 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238413fec0 con 0x7f23840a9920 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f237bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f2389d55640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:00.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:00.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.374+0000 7f238a556640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.575+0000 7f2389d55640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.575+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:00.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.575+0000 7f238a556640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:00.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.976+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:00.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.976+0000 7f2389d55640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:00.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:00.976+0000 7f238a556640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:01.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:01.777+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:01.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:01.777+0000 7f2389d55640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:01.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:01.777+0000 7f238a556640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient(hunting): tick 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f23840a9920 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f23840b45a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2384146450 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:03.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368010e20 con 0x7f2384146450 2024-03-25T18:19:03.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238406b100 con 0x7f23840b45a0 2024-03-25T18:19:03.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23840b2940 con 0x7f23840a9920 2024-03-25T18:19:03.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f237bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:03.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f236800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:03.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:03.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.374+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:03.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.575+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f236800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:03.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.575+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:03.575 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.575+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:03.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.976+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:03.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.976+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f236800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:03.976 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:03.976+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:04.710 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:00:07.328 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:04.727 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:00:07.389 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:04.727 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:00:07.389 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:04.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:04.777+0000 7f238ad57640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f236800f690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6789/0 2024-03-25T18:19:04.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:04.777+0000 7f238a556640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.22:6790/0 2024-03-25T18:19:04.777 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:04.777+0000 7f2389d55640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.33:6789/0 2024-03-25T18:19:05.036 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:00:07.654 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.054 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:00:07.716 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.054 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:00:07.717 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:06.374 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient(hunting): tick 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 1 -- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 1 -- >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f236800f690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f23840a9920 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2384146450 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f23840b45a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:06.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238413fec0 con 0x7f2384146450 2024-03-25T18:19:06.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368010e20 con 0x7f23840b45a0 2024-03-25T18:19:06.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f238406b100 con 0x7f23840a9920 2024-03-25T18:19:06.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.374+0000 7f237bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:06.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.537+0000 7f238ad57640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 0x7f23680056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:35424/0 (socket says 172.21.15.22:35424) 2024-03-25T18:19:06.537 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.537+0000 7f238ad57640 1 -- 172.21.15.22:0/3635524289 learned_addr learned my addr 172.21.15.22:0/3635524289 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:06.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.769+0000 7f238ad57640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:19:06.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.769+0000 7f238ad57640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:19:06.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.769+0000 7f238ad57640 1 --1- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 0x7f23680056d0 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T18:19:06.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.769+0000 7f238ad57640 1 --1- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 0x7f23680056d0 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:19:06.769 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.769+0000 7f2389554640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.33:6789/0 2024-03-25T18:19:06.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.793+0000 7f238a556640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:19:06.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.793+0000 7f238a556640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:19:06.793 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.793+0000 7f238a556640 1 --1- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 0x7f236800fdf0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-25T18:19:06.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.793+0000 7f238a556640 1 --1- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 0x7f236800fdf0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:19:06.794 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.793+0000 7f2389554640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.22:6790/0 2024-03-25T18:19:06.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.796+0000 7f2389d55640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368010560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:19:06.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.796+0000 7f2389d55640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368010560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:19:06.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.796+0000 7f2389d55640 1 --1- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 0x7f2368010560 :-1 s=OPENED pgs=2 cs=1 l=1).handle_message read tag failed 2024-03-25T18:19:06.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.796+0000 7f2389d55640 1 --1- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 0x7f2368010560 :-1 s=OPENED pgs=2 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:19:06.796 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:06.796+0000 7f2389554640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.22:6789/0 2024-03-25T18:19:09.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient(hunting): tick 2024-03-25T18:19:09.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:09.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:09.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:09.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368010560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f2384146450 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f23840b45a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f23840a9920 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23840b2940 con 0x7f2384146450 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368008eb0 con 0x7f23840b45a0 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368009040 con 0x7f23840a9920 2024-03-25T18:19:09.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:09.374+0000 7f237bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:12.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient(hunting): tick 2024-03-25T18:19:12.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:12.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:12.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23840a9920 legacy=0x7f236800fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23840b45a0 legacy=0x7f23680056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f2384146450 legacy=0x7f2368010540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f236800d660 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f23680127b0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f23680162c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368009220 con 0x7f236800d660 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368004770 con 0x7f23680162c0 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236800fdd0 con 0x7f23680127b0 2024-03-25T18:19:12.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:12.375+0000 7f237bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient(hunting): tick 2024-03-25T18:19:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 1 monclient(hunting): continuing hunt 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23680127b0 legacy=0x7f2368014ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23680162c0 legacy=0x7f2368018750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f236800d660 legacy=0x7f2368011090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f236800ff60 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f23680049a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f23680127b0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23680059b0 con 0x7f236800ff60 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368005490 con 0x7f23680049a0 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2368005670 con 0x7f23680127b0 2024-03-25T18:19:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.374+0000 7f237bfff640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:15.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.714+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480516592 0 0) 0x7f23680059b0 con 0x7f236800ff60 2024-03-25T18:19:15.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.714+0000 7f2389554640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.714+0000 7f2389554640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.714+0000 7f2389554640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.714+0000 7f2389554640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:19:15.715 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.714+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2370003520 con 0x7f236800ff60 2024-03-25T18:19:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.746+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668730417 0 0) 0x7f2368005490 con 0x7f23680049a0 2024-03-25T18:19:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.746+0000 7f2389554640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.746+0000 7f2389554640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.746+0000 7f2389554640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.746+0000 7f2389554640 10 monclient(hunting): my global_id is 4102 2024-03-25T18:19:15.747 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.746+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23680059b0 con 0x7f23680049a0 2024-03-25T18:19:15.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.748+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111205159 0 0) 0x7f2368005670 con 0x7f23680127b0 2024-03-25T18:19:15.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.748+0000 7f2389554640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.748+0000 7f2389554640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.748+0000 7f2389554640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.748+0000 7f2389554640 10 monclient(hunting): my global_id is 4100 2024-03-25T18:19:15.749 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.748+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2368005490 con 0x7f23680127b0 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637454812 0 0) 0x7f2370003520 con 0x7f236800ff60 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6790/0 conn(0x7f23680127b0 legacy=0x7f2368007e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.33:6789/0 conn(0x7f23680049a0 legacy=0x7f2368004d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 10 monclient: _finish_hunting 0 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 1 monclient: found mon.noname-a 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2368015bb0 con 0x7f236800ff60 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 10 monclient: _finish_auth 0 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 10 monclient: _check_auth_tickets 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2380003f70 con 0x7f236800ff60 2024-03-25T18:19:15.809 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f238c7ba640 5 monclient: authenticate success, global_id 4104 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f2389554640 10 monclient: dump: 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:15.810 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.807+0000 7f238c7ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23800040d0 con 0x7f236800ff60 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f2389554640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f2389554640 1 -- 172.21.15.22:0/3635524289 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2380004520 con 0x7f236800ff60 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f2389554640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f2389554640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f2389554640 10 monclient: dump: 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.856 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:15.857 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f238c7ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:15.857 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.854+0000 7f238c7ba640 4 set_mon_vals no callback set 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.855+0000 7f238c7ba640 10 monclient: shutdown 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.855+0000 7f238c7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.855+0000 7f238c7ba640 1 -- 172.21.15.22:0/3635524289 >> v1:172.21.15.22:6789/0 conn(0x7f236800ff60 legacy=0x7f2368010520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.855+0000 7f238c7ba640 1 -- 172.21.15.22:0/3635524289 shutdown_connections 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.855+0000 7f238c7ba640 1 -- 172.21.15.22:0/3635524289 >> 172.21.15.22:0/3635524289 conn(0x7f23840b7f40 msgr2=0x7f23840b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.855+0000 7f238c7ba640 1 -- 172.21.15.22:0/3635524289 shutdown_connections 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.856+0000 7f238c7ba640 1 -- 172.21.15.22:0/3635524289 wait complete. 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.856+0000 7f238c7ba640 10 monclient: build_initial_monmap 2024-03-25T18:19:15.859 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.856+0000 7f238c7ba640 10 monclient: monmap: 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 librados: starting msgr at 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 librados: starting objecter 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 Processor -- start 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 -- start start 2024-03-25T18:19:15.860 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 librados: setting wanted keys 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 librados: calling monclient init 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient: init 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient(hunting): picked mon.noname-a con 0x7f23680049a0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient(hunting): picked mon.noname-b con 0x7f236800ff60 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient(hunting): picked mon.noname-c con 0x7f23680127b0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2384144100 con 0x7f23680049a0 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23841438c0 con 0x7f236800ff60 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2384144590 con 0x7f23680127b0 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:15.861 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.857+0000 7f238c7ba640 10 monclient(hunting): authenticate will time out at 7155.081472s 2024-03-25T18:19:15.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.858+0000 7f238ad57640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f23680127b0 0x7f23841e21b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54152/0 (socket says 172.21.15.22:54152) 2024-03-25T18:19:15.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.858+0000 7f238ad57640 1 -- 172.21.15.22:0/1011246032 learned_addr learned my addr 172.21.15.22:0/1011246032 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:15.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.859+0000 7f237affd640 10 client.?.objecter ms_handle_connect 0x7f23680127b0 2024-03-25T18:19:15.862 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.859+0000 7f237affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.866+0000 7f237affd640 10 client.?.objecter ms_handle_connect 0x7f23680049a0 2024-03-25T18:19:15.867 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.866+0000 7f237affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.882+0000 7f237affd640 10 client.?.objecter ms_handle_connect 0x7f236800ff60 2024-03-25T18:19:15.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.882+0000 7f237affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.884+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324403274 0 0) 0x7f2384144100 con 0x7f23680049a0 2024-03-25T18:19:15.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.884+0000 7f237affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.884+0000 7f237affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.884+0000 7f237affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.884+0000 7f237affd640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:19:15.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.884+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2368005670 con 0x7f23680049a0 2024-03-25T18:19:15.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.894+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61677309 0 0) 0x7f2384144590 con 0x7f23680127b0 2024-03-25T18:19:15.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.894+0000 7f237affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.894+0000 7f237affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.894+0000 7f237affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.894+0000 7f237affd640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:19:15.895 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.894+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2384144100 con 0x7f23680127b0 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.913+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2749438874 0 0) 0x7f2368005670 con 0x7f23680049a0 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.913+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2384144590 con 0x7f23680049a0 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.913+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f23800044e0 con 0x7f23680049a0 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.913+0000 7f237affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.913+0000 7f237affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.913+0000 7f237affd640 10 monclient(hunting): dump: 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.914 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:15.915 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:15.915 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:15.915 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:15.915 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:15.915 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.913+0000 7f237affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.964+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1006129625 0 0) 0x7f2384144590 con 0x7f23680049a0 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.964+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 >> v1:172.21.15.22:6790/0 conn(0x7f23680127b0 legacy=0x7f23841e21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 >> v1:172.21.15.33:6789/0 conn(0x7f236800ff60 legacy=0x7f23841d7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 1 monclient: found mon.a 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:15.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2384145610 con 0x7f23680049a0 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f237affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 5 monclient: authenticate success, global_id 4113 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 10 monclient: _renew_subs 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2384143a90 con 0x7f23680049a0 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 10 monclient: _renew_subs 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23841d7e10 con 0x7f23680049a0 2024-03-25T18:19:15.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.965+0000 7f238c7ba640 1 librados: init done 2024-03-25T18:19:15.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.966+0000 7f238c7ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.966+0000 7f238c7ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.966+0000 7f238c7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:15.968 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:15.966+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23840ab430 con 0x7f23680049a0 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2380004970 con 0x7f23680049a0 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2380004d40 con 0x7f23680049a0 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 10 monclient: dump: 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:16.177 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f23800052f0 con 0x7f23680049a0 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f2388d53640 4 set_mon_vals no callback set 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1936577117 0 0) 0x7f2380006450 con 0x7f23680049a0 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 10 client.4113.objecter ms_dispatch 0x7f2384000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2380006ae0 con 0x7f23680049a0 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.178 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.175+0000 7f237affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:16.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.261+0000 7f238c7ba640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:19:16.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.261+0000 7f238c7ba640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:19:16.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.261+0000 7f238c7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:16.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.261+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f23840ab430 con 0x7f23680049a0 2024-03-25T18:19:16.401 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:16.399+0000 7f0efd02b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:19:16.409 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:16.408+0000 7fbf33610040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:19:16.456 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:16.455+0000 7f0efd02b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:19:16.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.467+0000 7f237affd640 1 -- 172.21.15.22:0/1011246032 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f2380005900 con 0x7f23680049a0 2024-03-25T18:19:16.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.467+0000 7f237affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:19:16.468 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.467+0000 7f237affd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:19:16.468 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2024-03-25T18:19:16.469 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:16.467+0000 7fbf33610040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.467+0000 7f238c7ba640 10 librados: watch_flush enter 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.467+0000 7f238c7ba640 10 librados: watch_flush exit 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 10 client.4113.objecter successfully canceled tick 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 10 monclient: shutdown 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 >> v1:172.21.15.22:6789/0 conn(0x7f23680049a0 legacy=0x7f23841431b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 shutdown_connections 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 >> 172.21.15.22:0/1011246032 conn(0x7f23840b7f40 msgr2=0x7f23840b6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 shutdown_connections 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 1 -- 172.21.15.22:0/1011246032 wait complete. 2024-03-25T18:19:16.470 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:16.468+0000 7f238c7ba640 1 librados: shutdown 2024-03-25T18:19:16.483 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:19:16.484 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:19:16.484 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:19:16.484 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:19:16.511 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:19:16.511 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:19:16.581 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:19:16.581 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:19:16.648 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-25T18:19:16.648 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:19:16.716 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:19:16.716 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:19:16.744 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:19:16.744 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:19:16.812 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:19:16.812 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:19:16.879 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-03-25T18:19:16.879 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:19:16.946 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 6253bf71-82c4-48c3-86bf-f3cbff91fb52 0 2024-03-25T18:19:17.020 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.019+0000 7f0efd02b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:19:17.049 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.048+0000 7fbf33610040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:19:17.078 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.077+0000 7f0efd02b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:19:17.087 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.086+0000 7f28f2893640 10 monclient: get_monmap_and_config 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.086+0000 7f28f2893640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.086+0000 7f28f2893640 10 monclient: monmap: 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:17.088040+0000 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:17.088040+0000 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:17.088 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.087+0000 7f28f2893640 1 Processor -- start 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.087+0000 7f28f2893640 1 -- start start 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.087+0000 7f28f2893640 10 monclient: init 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 10 monclient(hunting): picked mon.noname-a con 0x7f28ec0b4700 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 10 monclient(hunting): picked mon.noname-b con 0x7f28ec142a20 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 10 monclient(hunting): picked mon.noname-c con 0x7f28ec0a96b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec0a9d60 con 0x7f28ec0b4700 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec13ff10 con 0x7f28ec0a96b0 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec0aa1f0 con 0x7f28ec142a20 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.090 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28f2893640 10 monclient(hunting): authenticate will time out at 7176.252875s 2024-03-25T18:19:17.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28ebfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f28ec0b4700 0x7f28ec142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46364/0 (socket says 172.21.15.33:46364) 2024-03-25T18:19:17.091 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.088+0000 7f28ebfff640 1 -- 172.21.15.33:0/3182179942 learned_addr learned my addr 172.21.15.33:0/3182179942 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:17.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667269737 0 0) 0x7f28ec13ff10 con 0x7f28ec0a96b0 2024-03-25T18:19:17.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.092 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d8003520 con 0x7f28ec0a96b0 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055067771 0 0) 0x7f28ec0aa1f0 con 0x7f28ec142a20 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ec13ff10 con 0x7f28ec142a20 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333444576 0 0) 0x7f28ec0a9d60 con 0x7f28ec0b4700 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:19:17.093 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.091+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ec0aa1f0 con 0x7f28ec0b4700 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329335272 0 0) 0x7f28ec0aa1f0 con 0x7f28ec0b4700 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 >> v1:172.21.15.22:6790/0 conn(0x7f28ec142a20 legacy=0x7f28ec13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 >> v1:172.21.15.33:6789/0 conn(0x7f28ec0a96b0 legacy=0x7f28ec13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 1 monclient: found mon.noname-a 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec0b4ae0 con 0x7f28ec0b4700 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f28dc0015b0 con 0x7f28ec0b4700 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:17.096 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28eaffd640 10 monclient: dump: 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28f2893640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:19:17.097 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.094+0000 7f28f2893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:17.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28dc001710 con 0x7f28ec0b4700 2024-03-25T18:19:17.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28eaffd640 1 -- 172.21.15.33:0/3182179942 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f28dc001b60 con 0x7f28ec0b4700 2024-03-25T18:19:17.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.098 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28eaffd640 10 monclient: dump: 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28f2893640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.096+0000 7f28f2893640 4 set_mon_vals no callback set 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.097+0000 7f28f2893640 10 monclient: shutdown 2024-03-25T18:19:17.099 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.097+0000 7f28f2893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.097+0000 7f28f2893640 1 -- 172.21.15.33:0/3182179942 >> v1:172.21.15.22:6789/0 conn(0x7f28ec0b4700 legacy=0x7f28ec142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.097+0000 7f28f2893640 1 -- 172.21.15.33:0/3182179942 shutdown_connections 2024-03-25T18:19:17.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.097+0000 7f28f2893640 1 -- 172.21.15.33:0/3182179942 >> 172.21.15.33:0/3182179942 conn(0x7f28ec0668e0 msgr2=0x7f28ec066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.097+0000 7f28f2893640 1 -- 172.21.15.33:0/3182179942 shutdown_connections 2024-03-25T18:19:17.100 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.097+0000 7f28f2893640 1 -- 172.21.15.33:0/3182179942 wait complete. 2024-03-25T18:19:17.101 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.098+0000 7f28f2893640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.098+0000 7f28f2893640 10 monclient: monmap: 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.099+0000 7f28f2893640 1 librados: starting msgr at 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.099+0000 7f28f2893640 1 librados: starting objecter 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.099+0000 7f28f2893640 1 Processor -- start 2024-03-25T18:19:17.102 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.099+0000 7f28f2893640 1 -- start start 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.099+0000 7f28f2893640 1 librados: setting wanted keys 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.099+0000 7f28f2893640 1 librados: calling monclient init 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.099+0000 7f28f2893640 10 monclient: init 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 10 monclient(hunting): picked mon.noname-b con 0x7f28ec0a96b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 10 monclient(hunting): picked mon.noname-c con 0x7f28ec0b4700 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 10 monclient(hunting): picked mon.noname-a con 0x7f28ec142a20 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec138bd0 con 0x7f28ec142a20 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec1398a0 con 0x7f28ec0a96b0 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec139b80 con 0x7f28ec0b4700 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f2893640 10 monclient(hunting): authenticate will time out at 7176.265062s 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f0e30640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f28ec142a20 0x7f28ec1e26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46374/0 (socket says 172.21.15.33:46374) 2024-03-25T18:19:17.103 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.100+0000 7f28f0e30640 1 -- 172.21.15.33:0/4119703036 learned_addr learned my addr 172.21.15.33:0/4119703036 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:17.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.101+0000 7f28e8ff9640 10 client.?.objecter ms_handle_connect 0x7f28ec142a20 2024-03-25T18:19:17.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.101+0000 7f28e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.103+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687979109 0 0) 0x7f28ec138bd0 con 0x7f28ec142a20 2024-03-25T18:19:17.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.103+0000 7f28e8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.103+0000 7f28e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.104 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.103+0000 7f28e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.103+0000 7f28e8ff9640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:19:17.105 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.103+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28cc004eb0 con 0x7f28ec142a20 2024-03-25T18:19:17.105 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.104+0000 7fbf33610040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.107+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1730138704 0 0) 0x7f28cc004eb0 con 0x7f28ec142a20 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.107+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28ec138bd0 con 0x7f28ec142a20 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.107+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f28e4001c30 con 0x7f28ec142a20 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.107+0000 7f28e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.107+0000 7f28e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.107+0000 7f28e8ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.109 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.110 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.110 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.110 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.110 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.110 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.107+0000 7f28e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2084382252 0 0) 0x7f28ec138bd0 con 0x7f28ec142a20 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 >> v1:172.21.15.22:6790/0 conn(0x7f28ec0b4700 legacy=0x7f28ec130650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 >> v1:172.21.15.33:6789/0 conn(0x7f28ec0a96b0 legacy=0x7f28ec12ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 1 monclient: found mon.a 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec139e60 con 0x7f28ec142a20 2024-03-25T18:19:17.113 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 10 monclient: _renew_subs 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ec1385d0 con 0x7f28ec142a20 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 10 monclient: _renew_subs 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec137a70 con 0x7f28ec142a20 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.111+0000 7f28f2893640 1 librados: init done 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.112+0000 7f28f2893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.112+0000 7f28f2893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.112+0000 7f28f2893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.114 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.112+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec04f730 con 0x7f28ec142a20 2024-03-25T18:19:17.116 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28e4003990 con 0x7f28ec142a20 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f28e4001d90 con 0x7f28ec142a20 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 10 monclient: dump: 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.117 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e9ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f28e4004230 con 0x7f28ec142a20 2024-03-25T18:19:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 315200971 0 0) 0x7f28e4006810 con 0x7f28ec142a20 2024-03-25T18:19:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 10 client.4119.objecter ms_dispatch 0x7f28ec000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:19:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:19:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:19:17.118 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.115+0000 7f28e8ff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:19:17.127 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.126+0000 7f0efd02b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:19:17.152 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.151+0000 7fbf33610040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:19:17.183 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.182+0000 7f0efd02b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:19:17.208 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.207+0000 7fbf33610040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:19:17.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.215+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f28e40044e0 con 0x7f28ec142a20 2024-03-25T18:19:17.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.215+0000 7f28e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.216 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.215+0000 7f28e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:17.228 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.227+0000 7f0efd02b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:19:17.253 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.252+0000 7fbf33610040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:19:17.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.289+0000 7f28f2893640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6253bf71-82c4-48c3-86bf-f3cbff91fb52", "id": 0}] 2024-03-25T18:19:17.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.289+0000 7f28f2893640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6253bf71-82c4-48c3-86bf-f3cbff91fb52", "id": 0}] 2024-03-25T18:19:17.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.289+0000 7f28f2893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.290 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.289+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6253bf71-82c4-48c3-86bf-f3cbff91fb52", "id": 0} v 0) v1 -- 0x7f28ec13e650 con 0x7f28ec142a20 2024-03-25T18:19:17.347 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.346+0000 7f0efd02b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:19:17.374 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.373+0000 7fbf33610040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:19:17.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.398+0000 7f28e8ff9640 1 -- 172.21.15.33:0/4119703036 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6253bf71-82c4-48c3-86bf-f3cbff91fb52", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2644651492 0 732707515) 0x7f28e4005b40 con 0x7f28ec142a20 2024-03-25T18:19:17.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.398+0000 7f28e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6253bf71-82c4-48c3-86bf-f3cbff91fb52", "id": 0}] 2024-03-25T18:19:17.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.398+0000 7f28e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:17.399 INFO:teuthology.orchestra.run.smithi033.stdout:0 2024-03-25T18:19:17.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.398+0000 7f28f2893640 10 librados: watch_flush enter 2024-03-25T18:19:17.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.398+0000 7f28f2893640 10 librados: watch_flush exit 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 10 monclient: shutdown 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 >> v1:172.21.15.22:6789/0 conn(0x7f28ec142a20 legacy=0x7f28ec1e26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 shutdown_connections 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 >> 172.21.15.33:0/4119703036 conn(0x7f28ec0668e0 msgr2=0x7f28ec0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 shutdown_connections 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 1 -- 172.21.15.33:0/4119703036 wait complete. 2024-03-25T18:19:17.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.399+0000 7f28f2893640 1 librados: shutdown 2024-03-25T18:19:17.413 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70 1 2024-03-25T18:19:17.422 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.421+0000 7f0efd02b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:17.449 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.448+0000 7fbf33610040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:17.471 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.470+0000 7f0efd02b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:19:17.497 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.496+0000 7fbf33610040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:19:17.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.520+0000 7fa80b8fb640 10 monclient: get_monmap_and_config 2024-03-25T18:19:17.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.520+0000 7fa80b8fb640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.521 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.520+0000 7fa80b8fb640 10 monclient: monmap: 2024-03-25T18:19:17.521 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:17.521948+0000 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:17.521948+0000 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:17.522 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.521+0000 7fa80b8fb640 1 Processor -- start 2024-03-25T18:19:17.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.521+0000 7fa80b8fb640 1 -- start start 2024-03-25T18:19:17.523 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.521+0000 7fa80b8fb640 10 monclient: init 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.521+0000 7fa80b8fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.521+0000 7fa80b8fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa804070210 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa8040b06d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa8040705f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8040b0b80 con 0x7fa8040b06d0 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8040b0430 con 0x7fa8040705f0 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa804071670 con 0x7fa804070210 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa80b8fb640 10 monclient(hunting): authenticate will time out at 7176.686866s 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa808e96640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa8040b06d0 0x7fa8040afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46376/0 (socket says 172.21.15.33:46376) 2024-03-25T18:19:17.524 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.522+0000 7fa808e96640 1 -- 172.21.15.33:0/3400896920 learned_addr learned my addr 172.21.15.33:0/3400896920 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:17.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553780359 0 0) 0x7fa7fc001490 con 0x7fa8040705f0 2024-03-25T18:19:17.526 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f0003520 con 0x7fa8040705f0 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179788465 0 0) 0x7fa8040b0b80 con 0x7fa8040b06d0 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc001490 con 0x7fa8040b06d0 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958028903 0 0) 0x7fa804071670 con 0x7fa804070210 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:19:17.527 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.525+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8040b0b80 con 0x7fa804070210 2024-03-25T18:19:17.528 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.526+0000 7f0efd02b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3913022238 0 0) 0x7fa7f0003520 con 0x7fa8040705f0 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 >> v1:172.21.15.22:6790/0 conn(0x7fa804070210 legacy=0x7fa804140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 >> v1:172.21.15.22:6789/0 conn(0x7fa8040b06d0 legacy=0x7fa8040afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 1 monclient: found mon.noname-c 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa804070a60 con 0x7fa8040705f0 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.530 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa80b8fb640 5 monclient: authenticate success, global_id 4111 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa7fc001a00 con 0x7fa8040705f0 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa80b8fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa7fbfff640 10 monclient: dump: 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.531 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.532 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.532 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.528+0000 7fa80b8fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:17.552 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.551+0000 7fbf33610040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7fc001b60 con 0x7fa8040705f0 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa7fbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa7fbfff640 1 -- 172.21.15.33:0/3400896920 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa7fc001ef0 con 0x7fa8040705f0 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa7fbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa7fbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa7fbfff640 10 monclient: dump: 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.558 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa80b8fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:17.559 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.557+0000 7fa80b8fb640 4 set_mon_vals no callback set 2024-03-25T18:19:17.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.558+0000 7fa80b8fb640 10 monclient: shutdown 2024-03-25T18:19:17.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.558+0000 7fa80b8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.558+0000 7fa80b8fb640 1 -- 172.21.15.33:0/3400896920 >> v1:172.21.15.33:6789/0 conn(0x7fa8040705f0 legacy=0x7fa80414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.560 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.558+0000 7fa80b8fb640 1 -- 172.21.15.33:0/3400896920 shutdown_connections 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.558+0000 7fa80b8fb640 1 -- 172.21.15.33:0/3400896920 >> 172.21.15.33:0/3400896920 conn(0x7fa8040b5cd0 msgr2=0x7fa8040b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.558+0000 7fa80b8fb640 1 -- 172.21.15.33:0/3400896920 shutdown_connections 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.558+0000 7fa80b8fb640 1 -- 172.21.15.33:0/3400896920 wait complete. 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 10 monclient: monmap: 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 1 librados: starting msgr at 2024-03-25T18:19:17.561 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 1 librados: starting objecter 2024-03-25T18:19:17.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 1 Processor -- start 2024-03-25T18:19:17.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 1 -- start start 2024-03-25T18:19:17.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 1 librados: setting wanted keys 2024-03-25T18:19:17.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 1 librados: calling monclient init 2024-03-25T18:19:17.562 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.559+0000 7fa80b8fb640 10 monclient: init 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 10 monclient(hunting): picked mon.noname-b con 0x7fa804070210 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 10 monclient(hunting): picked mon.noname-a con 0x7fa8040705f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 10 monclient(hunting): picked mon.noname-c con 0x7fa8040b06d0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80413a750 con 0x7fa8040705f0 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80413b8a0 con 0x7fa804070210 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80413b2f0 con 0x7fa8040b06d0 2024-03-25T18:19:17.563 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa80b8fb640 10 monclient(hunting): authenticate will time out at 7176.724916s 2024-03-25T18:19:17.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa808e96640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa8040705f0 0x7fa8041d7ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46382/0 (socket says 172.21.15.33:46382) 2024-03-25T18:19:17.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa808e96640 1 -- 172.21.15.33:0/4078095043 learned_addr learned my addr 172.21.15.33:0/4078095043 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:17.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa7f9ffb640 10 client.?.objecter ms_handle_connect 0x7fa8040705f0 2024-03-25T18:19:17.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.560+0000 7fa7f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.561+0000 7fa7f9ffb640 10 client.?.objecter ms_handle_connect 0x7fa8040b06d0 2024-03-25T18:19:17.564 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.561+0000 7fa7f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 client.?.objecter ms_handle_connect 0x7fa804070210 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524613286 0 0) 0x7fa80413b2f0 con 0x7fa8040b06d0 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e8004eb0 con 0x7fa8040b06d0 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288360617 0 0) 0x7fa80413a750 con 0x7fa8040705f0 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:19:17.565 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.563+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa80413b2f0 con 0x7fa8040705f0 2024-03-25T18:19:17.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.565+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49689714 0 0) 0x7fa80413b8a0 con 0x7fa804070210 2024-03-25T18:19:17.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.565+0000 7fa7f9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.566 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.565+0000 7fa7f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.565+0000 7fa7f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.565+0000 7fa7f9ffb640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:19:17.567 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.565+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa80413a750 con 0x7fa804070210 2024-03-25T18:19:17.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.568+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3302150517 0 0) 0x7fa80413b2f0 con 0x7fa8040705f0 2024-03-25T18:19:17.569 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.568+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa80413b8a0 con 0x7fa8040705f0 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.568+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa7f4001950 con 0x7fa8040705f0 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.568+0000 7fa7f9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.568+0000 7fa7f9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.568+0000 7fa7f9ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.570 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.568+0000 7fa7f9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2905726220 0 0) 0x7fa80413b8a0 con 0x7fa8040705f0 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 >> v1:172.21.15.22:6790/0 conn(0x7fa8040b06d0 legacy=0x7fa804135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 >> v1:172.21.15.33:6789/0 conn(0x7fa804070210 legacy=0x7fa80413abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 1 monclient: found mon.a 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80413b480 con 0x7fa8040705f0 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa7f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.575 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 10 monclient: _renew_subs 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8041d81d0 con 0x7fa8040705f0 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 10 monclient: _renew_subs 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa804139030 con 0x7fa8040705f0 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.573+0000 7fa80b8fb640 1 librados: init done 2024-03-25T18:19:17.576 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.574+0000 7fa80b8fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.574+0000 7fa80b8fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.574+0000 7fa80b8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.577 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.574+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8041d8570 con 0x7fa8040705f0 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7f40047e0 con 0x7fa8040705f0 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa7f4004950 con 0x7fa8040705f0 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7f9ffb640 10 monclient: dump: 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.578 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa7f4004e40 con 0x7fa8040705f0 2024-03-25T18:19:17.579 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.576+0000 7fa7fb7fe640 4 set_mon_vals no callback set 2024-03-25T18:19:17.579 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.577+0000 7f0efd02b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:19:17.604 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:17.603+0000 7fbf33610040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 4001612862 0 0) 0x7fa7f4006310 con 0x7fa8040705f0 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 10 client.4125.objecter ms_dispatch 0x7fa804000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa7f40066e0 con 0x7fa8040705f0 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.679+0000 7fa7f9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.753+0000 7fa80b8fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70", "id": 1}] 2024-03-25T18:19:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.753+0000 7fa80b8fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70", "id": 1}] 2024-03-25T18:19:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.753+0000 7fa80b8fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.755 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.753+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70", "id": 1} v 0) v1 -- 0x7fa8041401d0 con 0x7fa8040705f0 2024-03-25T18:19:17.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.808+0000 7fa7f9ffb640 1 -- 172.21.15.33:0/4078095043 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1924481055 0 940483276) 0x7fa7f4006920 con 0x7fa8040705f0 2024-03-25T18:19:17.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.808+0000 7fa7f9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70", "id": 1}] 2024-03-25T18:19:17.810 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.808+0000 7fa7f9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:17.810 INFO:teuthology.orchestra.run.smithi033.stdout:1 2024-03-25T18:19:17.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 10 librados: watch_flush enter 2024-03-25T18:19:17.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 10 librados: watch_flush exit 2024-03-25T18:19:17.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:19:17.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:19:17.811 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 10 monclient: shutdown 2024-03-25T18:19:17.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 >> v1:172.21.15.22:6789/0 conn(0x7fa8040705f0 legacy=0x7fa8041d7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 shutdown_connections 2024-03-25T18:19:17.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 >> 172.21.15.33:0/4078095043 conn(0x7fa8040b5cd0 msgr2=0x7fa8040afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.809+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 shutdown_connections 2024-03-25T18:19:17.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.810+0000 7fa80b8fb640 1 -- 172.21.15.33:0/4078095043 wait complete. 2024-03-25T18:19:17.812 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.810+0000 7fa80b8fb640 1 librados: shutdown 2024-03-25T18:19:17.821 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 1268cb30-a412-4d09-8729-a87a96e89819 2 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.922+0000 7fd539a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.922+0000 7fd539a4a640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.922+0000 7fd539a4a640 10 monclient: monmap: 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:17.923296+0000 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:17.923296+0000 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:17.923 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.922+0000 7fd539a4a640 1 Processor -- start 2024-03-25T18:19:17.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.922+0000 7fd539a4a640 1 -- start start 2024-03-25T18:19:17.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.922+0000 7fd539a4a640 10 monclient: init 2024-03-25T18:19:17.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:17.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd5340b0530 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:17.925 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd5340b0910 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd534142800 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5340719f0 con 0x7fd5340b0910 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534070980 con 0x7fd5340b0530 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534072940 con 0x7fd534142800 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd539a4a640 10 monclient(hunting): authenticate will time out at 7177.088394s 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd52bfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fd5340b0910 0x7fd534071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46394/0 (socket says 172.21.15.33:46394) 2024-03-25T18:19:17.926 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.923+0000 7fd52bfff640 1 -- 172.21.15.33:0/3269117514 learned_addr learned my addr 172.21.15.33:0/3269117514 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:17.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869804074 0 0) 0x7fd534072940 con 0x7fd534142800 2024-03-25T18:19:17.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:19:17.927 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd520005730 con 0x7fd534142800 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098602517 0 0) 0x7fd534070980 con 0x7fd5340b0530 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534072940 con 0x7fd5340b0530 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467374598 0 0) 0x7fd5340719f0 con 0x7fd5340b0910 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.928 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:19:17.929 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.926+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534070980 con 0x7fd5340b0910 2024-03-25T18:19:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4179367009 0 0) 0x7fd520005730 con 0x7fd534142800 2024-03-25T18:19:17.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 >> v1:172.21.15.33:6789/0 conn(0x7fd5340b0530 legacy=0x7fd534070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 >> v1:172.21.15.22:6789/0 conn(0x7fd5340b0910 legacy=0x7fd534071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 1 monclient: found mon.noname-b 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534072520 con 0x7fd534142800 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd52c001a10 con 0x7fd534142800 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd5327fc640 10 monclient: dump: 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.932 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd539a4a640 5 monclient: authenticate success, global_id 4127 2024-03-25T18:19:17.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.930+0000 7fd539a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:17.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd52c001b70 con 0x7fd534142800 2024-03-25T18:19:17.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd5327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd5327fc640 1 -- 172.21.15.33:0/3269117514 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd52c001f80 con 0x7fd534142800 2024-03-25T18:19:17.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd5327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:17.958 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd5327fc640 10 monclient: dump: 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:17.959 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 4 set_mon_vals no callback set 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 10 monclient: shutdown 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 1 -- 172.21.15.33:0/3269117514 >> v1:172.21.15.22:6790/0 conn(0x7fd534142800 legacy=0x7fd534142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 1 -- 172.21.15.33:0/3269117514 shutdown_connections 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 1 -- 172.21.15.33:0/3269117514 >> 172.21.15.33:0/3269117514 conn(0x7fd53406bcb0 msgr2=0x7fd53406c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 1 -- 172.21.15.33:0/3269117514 shutdown_connections 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.957+0000 7fd539a4a640 1 -- 172.21.15.33:0/3269117514 wait complete. 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.958+0000 7fd539a4a640 10 monclient: build_initial_monmap 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.958+0000 7fd539a4a640 10 monclient: monmap: 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:17.962 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.958+0000 7fd539a4a640 1 librados: starting msgr at 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.958+0000 7fd539a4a640 1 librados: starting objecter 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.958+0000 7fd539a4a640 1 Processor -- start 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 1 -- start start 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 1 librados: setting wanted keys 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 1 librados: calling monclient init 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient: init 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd5340b0530 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:17.963 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd534142800 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd534138060 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534139060 con 0x7fd534142800 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534139300 con 0x7fd5340b0530 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd534138bc0 con 0x7fd534138060 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.959+0000 7fd539a4a640 10 monclient(hunting): authenticate will time out at 7177.124322s 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd52bfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fd534142800 0x7fd5341d65a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46398/0 (socket says 172.21.15.33:46398) 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd52bfff640 1 -- 172.21.15.33:0/890246802 learned_addr learned my addr 172.21.15.33:0/890246802 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd530ff9640 10 client.?.objecter ms_handle_connect 0x7fd534142800 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd530ff9640 10 client.?.objecter ms_handle_connect 0x7fd534138060 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd530ff9640 10 client.?.objecter ms_handle_connect 0x7fd5340b0530 2024-03-25T18:19:17.964 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.960+0000 7fd530ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:17.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691511917 0 0) 0x7fd534139060 con 0x7fd534142800 2024-03-25T18:19:17.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.965 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd514004eb0 con 0x7fd534142800 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570209485 0 0) 0x7fd534139300 con 0x7fd5340b0530 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.962+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534139060 con 0x7fd5340b0530 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.963+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233483914 0 0) 0x7fd534138bc0 con 0x7fd534138060 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.963+0000 7fd530ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.963+0000 7fd530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.963+0000 7fd530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.963+0000 7fd530ff9640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:19:17.966 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.963+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd534139300 con 0x7fd534138060 2024-03-25T18:19:17.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.967+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 966526618 0 0) 0x7fd534139060 con 0x7fd5340b0530 2024-03-25T18:19:17.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.967+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd534138bc0 con 0x7fd5340b0530 2024-03-25T18:19:17.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.967+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd524001930 con 0x7fd5340b0530 2024-03-25T18:19:17.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.967+0000 7fd530ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.967+0000 7fd530ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:17.968 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.967+0000 7fd530ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.969 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.967+0000 7fd530ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763613291 0 0) 0x7fd514004eb0 con 0x7fd534142800 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd534139060 con 0x7fd534142800 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd51c001910 con 0x7fd534142800 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.970 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2722822401 0 0) 0x7fd534139300 con 0x7fd534138060 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd514004eb0 con 0x7fd534138060 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd52c001e80 con 0x7fd534138060 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.968+0000 7fd530ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.971 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.972 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.972 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.972 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.972 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.972 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.972 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3283421029 0 0) 0x7fd534139060 con 0x7fd534142800 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 >> v1:172.21.15.22:6790/0 conn(0x7fd534138060 legacy=0x7fd5341d6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 >> v1:172.21.15.33:6789/0 conn(0x7fd5340b0530 legacy=0x7fd534135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 1 monclient: found mon.a 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd534138d50 con 0x7fd534142800 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd530ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:17.973 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 10 monclient: _renew_subs 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53413ef70 con 0x7fd534142800 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 10 monclient: _renew_subs 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5341364f0 con 0x7fd534142800 2024-03-25T18:19:17.974 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.971+0000 7fd539a4a640 1 librados: init done 2024-03-25T18:19:17.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.972+0000 7fd539a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.972+0000 7fd539a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.972+0000 7fd539a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:17.975 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.972+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd534139490 con 0x7fd534142800 2024-03-25T18:19:17.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd51c0047e0 con 0x7fd534142800 2024-03-25T18:19:17.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd51c004bf0 con 0x7fd534142800 2024-03-25T18:19:17.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:17.977 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 10 monclient: dump: 2024-03-25T18:19:17.977 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd51c0051a0 con 0x7fd534142800 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd531ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1583743927 0 0) 0x7fd51c006690 con 0x7fd534142800 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 10 client.4131.objecter ms_dispatch 0x7fd534000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:19:17.978 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:19:17.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:19:17.979 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:17.976+0000 7fd530ff9640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:19:17.988 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:17.987+0000 7f0efd02b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:19:18.011 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.010+0000 7fbf33610040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:19:18.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.077+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd51c006380 con 0x7fd534142800 2024-03-25T18:19:18.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.077+0000 7fd530ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.078 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.077+0000 7fd530ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:18.129 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.128+0000 7f0efd02b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:19:18.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.150+0000 7fd539a4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1268cb30-a412-4d09-8729-a87a96e89819", "id": 2}] 2024-03-25T18:19:18.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.150+0000 7fd539a4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1268cb30-a412-4d09-8729-a87a96e89819", "id": 2}] 2024-03-25T18:19:18.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.150+0000 7fd539a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.151 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.150+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1268cb30-a412-4d09-8729-a87a96e89819", "id": 2} v 0) v1 -- 0x7fd5340a9b40 con 0x7fd534142800 2024-03-25T18:19:18.152 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.151+0000 7fbf33610040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:19:18.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.179+0000 7fd530ff9640 1 -- 172.21.15.33:0/890246802 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1268cb30-a412-4d09-8729-a87a96e89819", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2013388304 0 216599125) 0x7fd51c005520 con 0x7fd534142800 2024-03-25T18:19:18.180 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.179+0000 7fd530ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1268cb30-a412-4d09-8729-a87a96e89819", "id": 2}] 2024-03-25T18:19:18.181 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.179+0000 7fd530ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:18.181 INFO:teuthology.orchestra.run.smithi033.stdout:2 2024-03-25T18:19:18.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 10 librados: watch_flush enter 2024-03-25T18:19:18.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 10 librados: watch_flush exit 2024-03-25T18:19:18.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:19:18.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:19:18.182 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 10 monclient: shutdown 2024-03-25T18:19:18.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 >> v1:172.21.15.22:6789/0 conn(0x7fd534142800 legacy=0x7fd5341d65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 shutdown_connections 2024-03-25T18:19:18.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 >> 172.21.15.33:0/890246802 conn(0x7fd53406bcb0 msgr2=0x7fd534074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 shutdown_connections 2024-03-25T18:19:18.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 1 -- 172.21.15.33:0/890246802 wait complete. 2024-03-25T18:19:18.183 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.180+0000 7fd539a4a640 1 librados: shutdown 2024-03-25T18:19:18.192 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new fe29129e-9772-43fd-9f2a-4251290662ca 3 2024-03-25T18:19:18.247 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.246+0000 7f0efd02b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:19:18.281 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.280+0000 7fbf33610040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.290+0000 7ff8f0e0f640 10 monclient: get_monmap_and_config 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.290+0000 7ff8f0e0f640 10 monclient: build_initial_monmap 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.290+0000 7ff8f0e0f640 10 monclient: monmap: 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:18.292166+0000 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:18.292166+0000 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:18.292 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.291+0000 7ff8f0e0f640 1 Processor -- start 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.291+0000 7ff8f0e0f640 1 -- start start 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.291+0000 7ff8f0e0f640 10 monclient: init 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ec070210 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ec0b06d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:18.294 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ec0705f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec0b0b80 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec0b0430 con 0x7ff8ec070210 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec071670 con 0x7ff8ec0705f0 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8f0e0f640 10 monclient(hunting): authenticate will time out at 7177.456992s 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8e3fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff8ec0b06d0 0x7ff8ec0afd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46402/0 (socket says 172.21.15.33:46402) 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8e3fff640 1 -- 172.21.15.33:0/512154086 learned_addr learned my addr 172.21.15.33:0/512154086 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:18.295 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.292+0000 7ff8ea59c640 1 --1- 172.21.15.33:0/512154086 >> v1:172.21.15.33:6789/0 conn(0x7ff8ec070210 0x7ff8ec140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:60626/0 (socket says 172.21.15.33:60626) 2024-03-25T18:19:18.296 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.295+0000 7f0efd02b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:19:18.296 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86690461 0 0) 0x7ff8dc001110 con 0x7ff8ec070210 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d8003520 con 0x7ff8ec070210 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062251557 0 0) 0x7ff8ec0b0b80 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc001110 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053486069 0 0) 0x7ff8ec071670 con 0x7ff8ec0705f0 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.297 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:19:18.298 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.295+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec0b0b80 con 0x7ff8ec0705f0 2024-03-25T18:19:18.299 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911716145 0 0) 0x7ff8d8003520 con 0x7ff8ec070210 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 >> v1:172.21.15.22:6790/0 conn(0x7ff8ec0705f0 legacy=0x7ff8ec14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 >> v1:172.21.15.22:6789/0 conn(0x7ff8ec0b06d0 legacy=0x7ff8ec0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 1 monclient: found mon.noname-c 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec070a60 con 0x7ff8ec070210 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8dc001680 con 0x7ff8ec070210 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:18.300 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8e9d9b640 10 monclient: dump: 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8f0e0f640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:19:18.301 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.298+0000 7ff8f0e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:18.302 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8dc0017e0 con 0x7ff8ec070210 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8e9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8e9d9b640 1 -- 172.21.15.33:0/512154086 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8dc001bb0 con 0x7ff8ec070210 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8e9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8e9d9b640 10 monclient: dump: 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.303 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8f0e0f640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.300+0000 7ff8f0e0f640 4 set_mon_vals no callback set 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.301+0000 7ff8f0e0f640 10 monclient: shutdown 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.301+0000 7ff8f0e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.301+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/512154086 >> v1:172.21.15.33:6789/0 conn(0x7ff8ec070210 legacy=0x7ff8ec140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.301+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/512154086 shutdown_connections 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.301+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/512154086 >> 172.21.15.33:0/512154086 conn(0x7ff8ec0b5cd0 msgr2=0x7ff8ec0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.301+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/512154086 shutdown_connections 2024-03-25T18:19:18.304 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.301+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/512154086 wait complete. 2024-03-25T18:19:18.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.302+0000 7ff8f0e0f640 10 monclient: build_initial_monmap 2024-03-25T18:19:18.306 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.302+0000 7ff8f0e0f640 10 monclient: monmap: 2024-03-25T18:19:18.306 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:18.306 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.302+0000 7ff8f0e0f640 1 librados: starting msgr at 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.302+0000 7ff8f0e0f640 1 librados: starting objecter 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 1 Processor -- start 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 1 -- start start 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 1 librados: setting wanted keys 2024-03-25T18:19:18.307 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 1 librados: calling monclient init 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient: init 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ec070210 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ec0705f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ec0b06d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec138730 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec1d9720 con 0x7ff8ec070210 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec13afd0 con 0x7ff8ec0705f0 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.303+0000 7ff8f0e0f640 10 monclient(hunting): authenticate will time out at 7177.468369s 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8ead9d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff8ec0b06d0 0x7ff8ec1d9010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46408/0 (socket says 172.21.15.33:46408) 2024-03-25T18:19:18.308 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8ead9d640 1 -- 172.21.15.33:0/4002347623 learned_addr learned my addr 172.21.15.33:0/4002347623 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:18.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8e2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8ec070210 2024-03-25T18:19:18.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8e2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8ec0b06d0 2024-03-25T18:19:18.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8e2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8ec0705f0 2024-03-25T18:19:18.309 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.304+0000 7ff8e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638780517 0 0) 0x7ff8ec138730 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8cc004eb0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547393994 0 0) 0x7ff8ec1d9720 con 0x7ff8ec070210 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.310 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec138730 con 0x7ff8ec070210 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401321116 0 0) 0x7ff8ec13afd0 con 0x7ff8ec0705f0 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:19:18.311 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.306+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec1d9720 con 0x7ff8ec0705f0 2024-03-25T18:19:18.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1386706590 0 0) 0x7ff8cc004eb0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ec13afd0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8e4001d10 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:18.312 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:18.312 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.313 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:18.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 702744767 0 0) 0x7ff8ec1d9720 con 0x7ff8ec0705f0 2024-03-25T18:19:18.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8cc004eb0 con 0x7ff8ec0705f0 2024-03-25T18:19:18.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8d4001770 con 0x7ff8ec0705f0 2024-03-25T18:19:18.314 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.311+0000 7ff8e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.312+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2332075964 0 0) 0x7ff8ec138730 con 0x7ff8ec070210 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.312+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ec1d9720 con 0x7ff8ec070210 2024-03-25T18:19:18.315 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.312+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8dc001000 con 0x7ff8ec070210 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.312+0000 7ff8e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.312+0000 7ff8e2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.312+0000 7ff8e2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.316 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2881250456 0 0) 0x7ff8ec13afd0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 >> v1:172.21.15.22:6790/0 conn(0x7ff8ec0705f0 legacy=0x7ff8ec1d8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 >> v1:172.21.15.33:6789/0 conn(0x7ff8ec070210 legacy=0x7ff8ec139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:18.318 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 1 monclient: found mon.a 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec139f40 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 10 monclient: _renew_subs 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec139840 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 10 monclient: _renew_subs 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.319 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec1367f0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.315+0000 7ff8f0e0f640 1 librados: init done 2024-03-25T18:19:18.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.316+0000 7ff8f0e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.316+0000 7ff8f0e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.316+0000 7ff8f0e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.316+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec0682a0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.320 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8e4003990 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8e4003d60 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 10 monclient: dump: 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.321 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff8e4004310 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e959a640 4 set_mon_vals no callback set 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3035832890 0 0) 0x7ff8e4006a30 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 10 client.4137.objecter ms_dispatch 0x7ff8ec000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:19:18.322 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.319+0000 7ff8e2ffd640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:19:18.329 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.328+0000 7fbf33610040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:19:18.416 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.414+0000 7f0efd02b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:18.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.417+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff8e40066b0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.417+0000 7ff8e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.418 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.417+0000 7ff8e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:18.446 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.445+0000 7fbf33610040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:18.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.491+0000 7ff8f0e0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fe29129e-9772-43fd-9f2a-4251290662ca", "id": 3}] 2024-03-25T18:19:18.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.491+0000 7ff8f0e0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fe29129e-9772-43fd-9f2a-4251290662ca", "id": 3}] 2024-03-25T18:19:18.492 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.491+0000 7ff8f0e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.493 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.491+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fe29129e-9772-43fd-9f2a-4251290662ca", "id": 3} v 0) v1 -- 0x7ff8ec0718d0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.515 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.514+0000 7f0efd02b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:19:18.542 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.541+0000 7ff8e2ffd640 1 -- 172.21.15.33:0/4002347623 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fe29129e-9772-43fd-9f2a-4251290662ca", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1218689818 0 525046306) 0x7ff8e40045c0 con 0x7ff8ec0b06d0 2024-03-25T18:19:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.541+0000 7ff8e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fe29129e-9772-43fd-9f2a-4251290662ca", "id": 3}] 2024-03-25T18:19:18.543 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.541+0000 7ff8e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:18.543 INFO:teuthology.orchestra.run.smithi033.stdout:3 2024-03-25T18:19:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 10 librados: watch_flush enter 2024-03-25T18:19:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 10 librados: watch_flush exit 2024-03-25T18:19:18.544 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 10 monclient: shutdown 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 >> v1:172.21.15.22:6789/0 conn(0x7ff8ec0b06d0 legacy=0x7ff8ec1d9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 shutdown_connections 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 >> 172.21.15.33:0/4002347623 conn(0x7ff8ec0b5cd0 msgr2=0x7ff8ec0afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 shutdown_connections 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 1 -- 172.21.15.33:0/4002347623 wait complete. 2024-03-25T18:19:18.545 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.542+0000 7ff8f0e0f640 1 librados: shutdown 2024-03-25T18:19:18.548 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.547+0000 7fbf33610040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:19:18.554 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0 4 2024-03-25T18:19:18.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.654+0000 7f77a13ba640 10 monclient: get_monmap_and_config 2024-03-25T18:19:18.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.654+0000 7f77a13ba640 10 monclient: build_initial_monmap 2024-03-25T18:19:18.655 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.654+0000 7f77a13ba640 10 monclient: monmap: 2024-03-25T18:19:18.655 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:18.655745+0000 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:18.655745+0000 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:18.656 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.657 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 1 Processor -- start 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 1 -- start start 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient: init 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient(hunting): picked mon.noname-b con 0x7f779c06c2d0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient(hunting): picked mon.noname-a con 0x7f779c06c6b0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient(hunting): picked mon.noname-c con 0x7f779c142d90 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0b0840 con 0x7f779c06c6b0 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c143200 con 0x7f779c142d90 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c06d810 con 0x7f779c06c2d0 2024-03-25T18:19:18.658 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.655+0000 7f77a13ba640 10 monclient(hunting): authenticate will time out at 7177.820500s 2024-03-25T18:19:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.656+0000 7f779a7fc640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f779c06c6b0 0x7f779c142680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46420/0 (socket says 172.21.15.33:46420) 2024-03-25T18:19:18.659 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.656+0000 7f779a7fc640 1 -- 172.21.15.33:0/2179068252 learned_addr learned my addr 172.21.15.33:0/2179068252 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:18.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563744866 0 0) 0x7f779c06d810 con 0x7f779c06c2d0 2024-03-25T18:19:18.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.660 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7784003520 con 0x7f779c06c2d0 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57397547 0 0) 0x7f779c0b0840 con 0x7f779c06c6b0 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c06d810 con 0x7f779c06c6b0 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205006930 0 0) 0x7f7794001490 con 0x7f779c142d90 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.661 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:19:18.662 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.659+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c0b0840 con 0x7f779c142d90 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1583761854 0 0) 0x7f779c06d810 con 0x7f779c06c6b0 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 >> v1:172.21.15.22:6790/0 conn(0x7f779c06c2d0 legacy=0x7f779c0afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 >> v1:172.21.15.33:6789/0 conn(0x7f779c142d90 legacy=0x7f779c14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 1 monclient: found mon.noname-a 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0ae260 con 0x7f779c06c6b0 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:18.664 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7790001670 con 0x7f779c06c6b0 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f77a13ba640 5 monclient: authenticate success, global_id 4140 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f77a13ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f7799ffb640 10 monclient: dump: 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.665 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.666 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.666 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.662+0000 7f77a13ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.663+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77900017d0 con 0x7f779c06c6b0 2024-03-25T18:19:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.663+0000 7f7799ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.663+0000 7f7799ffb640 1 -- 172.21.15.33:0/2179068252 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7790001b60 con 0x7f779c06c6b0 2024-03-25T18:19:18.669 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.663+0000 7f77a13ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.663+0000 7f77a13ba640 4 set_mon_vals no callback set 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.663+0000 7f7799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f7799ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f7799ffb640 10 monclient: dump: 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.670 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f77a13ba640 10 monclient: shutdown 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f77a13ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f77a13ba640 1 -- 172.21.15.33:0/2179068252 >> v1:172.21.15.22:6789/0 conn(0x7f779c06c6b0 legacy=0x7f779c142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f77a13ba640 1 -- 172.21.15.33:0/2179068252 shutdown_connections 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f77a13ba640 1 -- 172.21.15.33:0/2179068252 >> 172.21.15.33:0/2179068252 conn(0x7f779c0b5e40 msgr2=0x7f779c0b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f77a13ba640 1 -- 172.21.15.33:0/2179068252 shutdown_connections 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.664+0000 7f77a13ba640 1 -- 172.21.15.33:0/2179068252 wait complete. 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.665+0000 7f77a13ba640 10 monclient: build_initial_monmap 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.665+0000 7f77a13ba640 10 monclient: monmap: 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:18.671 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.665+0000 7f77a13ba640 1 librados: starting msgr at 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.665+0000 7f77a13ba640 1 librados: starting objecter 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.665+0000 7f77a13ba640 1 Processor -- start 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.665+0000 7f77a13ba640 1 -- start start 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 1 librados: setting wanted keys 2024-03-25T18:19:18.672 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 1 librados: calling monclient init 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient: init 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient(hunting): picked mon.noname-b con 0x7f779c06c2d0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient(hunting): picked mon.noname-a con 0x7f779c06c6b0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient(hunting): picked mon.noname-c con 0x7f779c142d90 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c143af0 con 0x7f779c06c6b0 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c1db8c0 con 0x7f779c06c2d0 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0715d0 con 0x7f779c142d90 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f77a13ba640 10 monclient(hunting): authenticate will time out at 7177.831292s 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f779a7fc640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f779c06c6b0 0x7f779c1daaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46426/0 (socket says 172.21.15.33:46426) 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.666+0000 7f779a7fc640 1 -- 172.21.15.33:0/1964608014 learned_addr learned my addr 172.21.15.33:0/1964608014 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.667+0000 7f77837fe640 10 client.?.objecter ms_handle_connect 0x7f779c06c6b0 2024-03-25T18:19:18.673 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.667+0000 7f77837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.667+0000 7f77837fe640 10 client.?.objecter ms_handle_connect 0x7f779c06c2d0 2024-03-25T18:19:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.667+0000 7f77837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.668+0000 7f77837fe640 10 client.?.objecter ms_handle_connect 0x7f779c142d90 2024-03-25T18:19:18.674 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.668+0000 7f77837fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086229112 0 0) 0x7f779c143af0 con 0x7f779c06c6b0 2024-03-25T18:19:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.675 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7788004eb0 con 0x7f779c06c6b0 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016384766 0 0) 0x7f779c1db8c0 con 0x7f779c06c2d0 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.669+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c143af0 con 0x7f779c06c2d0 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.670+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268221409 0 0) 0x7f779c0715d0 con 0x7f779c142d90 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.670+0000 7f77837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.670+0000 7f77837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:18.676 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.670+0000 7f77837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.670+0000 7f77837fe640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.670+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779c1db8c0 con 0x7f779c142d90 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.673+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1624960050 0 0) 0x7f7788004eb0 con 0x7f779c06c6b0 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.674+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f779c0715d0 con 0x7f779c06c6b0 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.674+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7790001ba0 con 0x7f779c06c6b0 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.674+0000 7f77837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.674+0000 7f77837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.674+0000 7f77837fe640 10 monclient(hunting): dump: 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.677 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.678 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.678 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.678 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.678 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.678 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.678 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.674+0000 7f77837fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1488352909 0 0) 0x7f779c143af0 con 0x7f779c06c2d0 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7788004eb0 con 0x7f779c06c2d0 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f778c001770 con 0x7f779c06c2d0 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 10 monclient(hunting): dump: 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.680 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1913080103 0 0) 0x7f779c1db8c0 con 0x7f779c142d90 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f779c143af0 con 0x7f779c142d90 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7794001c60 con 0x7f779c142d90 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.675+0000 7f77837fe640 10 monclient(hunting): dump: 2024-03-25T18:19:18.681 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1091766582 0 0) 0x7f779c0715d0 con 0x7f779c06c6b0 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 >> v1:172.21.15.22:6790/0 conn(0x7f779c142d90 legacy=0x7f779c1db1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 >> v1:172.21.15.33:6789/0 conn(0x7f779c06c2d0 legacy=0x7f779c070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:18.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 1 monclient: found mon.a 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0700b0 con 0x7f779c06c6b0 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 5 monclient: authenticate success, global_id 4143 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 10 monclient: _renew_subs 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c133da0 con 0x7f779c06c6b0 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 10 monclient: _renew_subs 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c13f9f0 con 0x7f779c06c6b0 2024-03-25T18:19:18.683 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.676+0000 7f77a13ba640 1 librados: init done 2024-03-25T18:19:18.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.677+0000 7f77a13ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.677+0000 7f77a13ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.677+0000 7f77a13ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.684 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.677+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c04f730 con 0x7f779c06c6b0 2024-03-25T18:19:18.687 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.686+0000 7f0efd02b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:18.716 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.714+0000 7fbf33610040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:19:18.731 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.730+0000 7f0efd02b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:18.760 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.759+0000 7fbf33610040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7790004a90 con 0x7f779c06c6b0 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f77837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7790004da0 con 0x7f779c06c6b0 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f77837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f77837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f77837fe640 10 monclient: dump: 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:18.783 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7790005290 con 0x7f779c06c6b0 2024-03-25T18:19:18.784 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.782+0000 7f7798ff9640 4 set_mon_vals no callback set 2024-03-25T18:19:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2984112511 0 0) 0x7f7790006860 con 0x7f779c06c6b0 2024-03-25T18:19:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 10 client.4143.objecter ms_dispatch 0x7f779c000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:19:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:19:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:19:18.785 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T18:19:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7790006bd0 con 0x7f779c06c6b0 2024-03-25T18:19:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:18.786 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.783+0000 7f77837fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:18.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.857+0000 7f77a13ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0", "id": 4}] 2024-03-25T18:19:18.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.857+0000 7f77a13ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0", "id": 4}] 2024-03-25T18:19:18.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.857+0000 7f77a13ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:18.858 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.857+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0", "id": 4} v 0) v1 -- 0x7f779c069f30 con 0x7f779c06c6b0 2024-03-25T18:19:18.892 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:18.891+0000 7f0efd02b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:19:18.925 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:18.924+0000 7fbf33610040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:19:18.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.929+0000 7f77837fe640 1 -- 172.21.15.33:0/1964608014 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1058867517 0 1697012583) 0x7f7790006420 con 0x7f779c06c6b0 2024-03-25T18:19:18.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.929+0000 7f77837fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0", "id": 4}] 2024-03-25T18:19:18.931 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.929+0000 7f77837fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:18.931 INFO:teuthology.orchestra.run.smithi033.stdout:4 2024-03-25T18:19:18.932 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 10 librados: watch_flush enter 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 10 librados: watch_flush exit 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 10 client.4143.objecter successfully canceled tick 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 10 monclient: shutdown 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 >> v1:172.21.15.22:6789/0 conn(0x7f779c06c6b0 legacy=0x7f779c1daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 shutdown_connections 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.930+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 >> 172.21.15.33:0/1964608014 conn(0x7f779c0b5e40 msgr2=0x7f779c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.931+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 shutdown_connections 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.931+0000 7f77a13ba640 1 -- 172.21.15.33:0/1964608014 wait complete. 2024-03-25T18:19:18.933 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:18.931+0000 7f77a13ba640 1 librados: shutdown 2024-03-25T18:19:18.943 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new ab490014-db2a-47a2-92f8-308d207490c0 5 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.042+0000 7f4e3a681640 10 monclient: get_monmap_and_config 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.042+0000 7f4e3a681640 10 monclient: build_initial_monmap 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.042+0000 7f4e3a681640 10 monclient: monmap: 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:19.044197+0000 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:19.044197+0000 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.044 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:19.045 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:19.045 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:19.045 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.043+0000 7f4e3a681640 1 Processor -- start 2024-03-25T18:19:19.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.043+0000 7f4e3a681640 1 -- start start 2024-03-25T18:19:19.046 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.043+0000 7f4e3a681640 10 monclient: init 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 10 monclient(hunting): picked mon.noname-c con 0x7f4e340b4700 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 10 monclient(hunting): picked mon.noname-b con 0x7f4e34142a20 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 10 monclient(hunting): picked mon.noname-a con 0x7f4e340a96b0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340a9d60 con 0x7f4e340a96b0 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3413ff10 con 0x7f4e340b4700 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340aa1f0 con 0x7f4e34142a20 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e3a681640 10 monclient(hunting): authenticate will time out at 7178.209146s 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e38c1e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f4e340a96b0 0x7f4e3413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46428/0 (socket says 172.21.15.33:46428) 2024-03-25T18:19:19.047 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.044+0000 7f4e38c1e640 1 -- 172.21.15.33:0/3995798269 learned_addr learned my addr 172.21.15.33:0/3995798269 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407316053 0 0) 0x7f4e340a9d60 con 0x7f4e340a96b0 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e20003520 con 0x7f4e340a96b0 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197209906 0 0) 0x7f4e3413ff10 con 0x7f4e340b4700 2024-03-25T18:19:19.049 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.047+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e340a9d60 con 0x7f4e340b4700 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.048+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216952714 0 0) 0x7f4e340aa1f0 con 0x7f4e34142a20 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.048+0000 7f4e32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.048+0000 7f4e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.048+0000 7f4e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.048+0000 7f4e32ffd640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:19:19.050 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.048+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3413ff10 con 0x7f4e34142a20 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686432732 0 0) 0x7f4e20003520 con 0x7f4e340a96b0 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 >> v1:172.21.15.22:6790/0 conn(0x7f4e34142a20 legacy=0x7f4e3413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 >> v1:172.21.15.33:6789/0 conn(0x7f4e340b4700 legacy=0x7f4e34142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 1 monclient: found mon.noname-a 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:19:19.052 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e340b4ae0 con 0x7f4e340a96b0 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.050+0000 7f4e32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.051+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e2c001a10 con 0x7f4e340a96b0 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.051+0000 7f4e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.051+0000 7f4e32ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.051+0000 7f4e32ffd640 10 monclient: dump: 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.053 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.054 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.054 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.054 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.054 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.051+0000 7f4e3a681640 5 monclient: authenticate success, global_id 4146 2024-03-25T18:19:19.054 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.051+0000 7f4e3a681640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:19.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e2c001b70 con 0x7f4e340a96b0 2024-03-25T18:19:19.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:19.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e32ffd640 1 -- 172.21.15.33:0/3995798269 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e2c001f80 con 0x7f4e340a96b0 2024-03-25T18:19:19.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.057 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e32ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e32ffd640 10 monclient: dump: 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.058 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e3a681640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.052+0000 7f4e3a681640 4 set_mon_vals no callback set 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.053+0000 7f4e3a681640 10 monclient: shutdown 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.053+0000 7f4e3a681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.053+0000 7f4e3a681640 1 -- 172.21.15.33:0/3995798269 >> v1:172.21.15.22:6789/0 conn(0x7f4e340a96b0 legacy=0x7f4e3413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.053+0000 7f4e3a681640 1 -- 172.21.15.33:0/3995798269 shutdown_connections 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.053+0000 7f4e3a681640 1 -- 172.21.15.33:0/3995798269 >> 172.21.15.33:0/3995798269 conn(0x7f4e340668e0 msgr2=0x7f4e34066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.053+0000 7f4e3a681640 1 -- 172.21.15.33:0/3995798269 shutdown_connections 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.053+0000 7f4e3a681640 1 -- 172.21.15.33:0/3995798269 wait complete. 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 10 monclient: build_initial_monmap 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 10 monclient: monmap: 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:19.059 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 1 librados: starting msgr at 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 1 librados: starting objecter 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 1 Processor -- start 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 1 -- start start 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 1 librados: setting wanted keys 2024-03-25T18:19:19.060 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 1 librados: calling monclient init 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.054+0000 7f4e3a681640 10 monclient: init 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 10 monclient(hunting): picked mon.noname-c con 0x7f4e340a96b0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 10 monclient(hunting): picked mon.noname-a con 0x7f4e340b4700 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 10 monclient(hunting): picked mon.noname-b con 0x7f4e34142a20 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340ac100 con 0x7f4e340b4700 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e340ae050 con 0x7f4e34142a20 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3414e6c0 con 0x7f4e340a96b0 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e3a681640 10 monclient(hunting): authenticate will time out at 7178.220037s 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e337fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f4e340b4700 0x7f4e3413a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46440/0 (socket says 172.21.15.33:46440) 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.055+0000 7f4e337fe640 1 -- 172.21.15.33:0/4167125706 learned_addr learned my addr 172.21.15.33:0/4167125706 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:19.061 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.056+0000 7f4e30ff9640 10 client.?.objecter ms_handle_connect 0x7f4e340b4700 2024-03-25T18:19:19.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.056+0000 7f4e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:19.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.056+0000 7f4e30ff9640 10 client.?.objecter ms_handle_connect 0x7f4e340a96b0 2024-03-25T18:19:19.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.056+0000 7f4e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:19.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.056+0000 7f4e30ff9640 10 client.?.objecter ms_handle_connect 0x7f4e34142a20 2024-03-25T18:19:19.062 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.056+0000 7f4e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846079521 0 0) 0x7f4e340ac100 con 0x7f4e340b4700 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e14004eb0 con 0x7f4e340b4700 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051953841 0 0) 0x7f4e340ae050 con 0x7f4e34142a20 2024-03-25T18:19:19.063 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e340ac100 con 0x7f4e34142a20 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387831898 0 0) 0x7f4e3414e6c0 con 0x7f4e340a96b0 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:19:19.064 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.058+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e340ae050 con 0x7f4e340a96b0 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.062+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3635023440 0 0) 0x7f4e340ac100 con 0x7f4e34142a20 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.062+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e3414e6c0 con 0x7f4e34142a20 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.062+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e2c001e80 con 0x7f4e34142a20 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.062+0000 7f4e30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.062+0000 7f4e30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.062+0000 7f4e30ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.066 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.062+0000 7f4e30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.063+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111184864 0 0) 0x7f4e14004eb0 con 0x7f4e340b4700 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.063+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e340ac100 con 0x7f4e340b4700 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.063+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e1c001a10 con 0x7f4e340b4700 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.063+0000 7f4e30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.063+0000 7f4e30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.063+0000 7f4e30ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.067 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.064+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 361512497 0 0) 0x7f4e340ae050 con 0x7f4e340a96b0 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.064+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e14004eb0 con 0x7f4e340a96b0 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.064+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e24001930 con 0x7f4e340a96b0 2024-03-25T18:19:19.068 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.064+0000 7f4e30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.064+0000 7f4e30ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.064+0000 7f4e30ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.069 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4221398489 0 0) 0x7f4e340ac100 con 0x7f4e340b4700 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 >> v1:172.21.15.22:6790/0 conn(0x7f4e340a96b0 legacy=0x7f4e340ae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 >> v1:172.21.15.33:6789/0 conn(0x7f4e34142a20 legacy=0x7f4e340ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 1 monclient: found mon.a 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3414e850 con 0x7f4e340b4700 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 5 monclient: authenticate success, global_id 4149 2024-03-25T18:19:19.071 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 10 monclient: _renew_subs 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e340ace10 con 0x7f4e340b4700 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 10 monclient: _renew_subs 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e341e3560 con 0x7f4e340b4700 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.066+0000 7f4e3a681640 1 librados: init done 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.067+0000 7f4e3a681640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.067+0000 7f4e3a681640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.067+0000 7f4e3a681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:19.072 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.067+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e340ac2b0 con 0x7f4e340b4700 2024-03-25T18:19:19.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e1c001c70 con 0x7f4e340b4700 2024-03-25T18:19:19.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:19.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e1c0038c0 con 0x7f4e340b4700 2024-03-25T18:19:19.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.169 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e30ff9640 10 monclient: dump: 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.170 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e327fc640 4 set_mon_vals no callback set 2024-03-25T18:19:19.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.168+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4e1c003e70 con 0x7f4e340b4700 2024-03-25T18:19:19.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3927371680 0 0) 0x7f4e1c0065b0 con 0x7f4e340b4700 2024-03-25T18:19:19.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 10 client.4149.objecter ms_dispatch 0x7f4e34000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:19:19.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:19:19.171 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:19:19.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T18:19:19.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4e1c004300 con 0x7f4e340b4700 2024-03-25T18:19:19.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:19.172 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.169+0000 7f4e30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:19.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.242+0000 7f4e3a681640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ab490014-db2a-47a2-92f8-308d207490c0", "id": 5}] 2024-03-25T18:19:19.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.242+0000 7f4e3a681640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab490014-db2a-47a2-92f8-308d207490c0", "id": 5}] 2024-03-25T18:19:19.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.242+0000 7f4e3a681640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:19.244 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.243+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ab490014-db2a-47a2-92f8-308d207490c0", "id": 5} v 0) v1 -- 0x7f4e3404f730 con 0x7f4e340b4700 2024-03-25T18:19:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.285+0000 7f4e30ff9640 1 -- 172.21.15.33:0/4167125706 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ab490014-db2a-47a2-92f8-308d207490c0", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2151477141 0 1988412176) 0x7f4e1c006030 con 0x7f4e340b4700 2024-03-25T18:19:19.286 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.285+0000 7f4e30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ab490014-db2a-47a2-92f8-308d207490c0", "id": 5}] 2024-03-25T18:19:19.287 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.285+0000 7f4e30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:19.287 INFO:teuthology.orchestra.run.smithi033.stdout:5 2024-03-25T18:19:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 10 librados: watch_flush enter 2024-03-25T18:19:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 10 librados: watch_flush exit 2024-03-25T18:19:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T18:19:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 10 client.4149.objecter successfully canceled tick 2024-03-25T18:19:19.288 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 10 monclient: shutdown 2024-03-25T18:19:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 >> v1:172.21.15.22:6789/0 conn(0x7f4e340b4700 legacy=0x7f4e3413a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 shutdown_connections 2024-03-25T18:19:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 >> 172.21.15.33:0/4167125706 conn(0x7f4e340668e0 msgr2=0x7f4e34066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 shutdown_connections 2024-03-25T18:19:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 1 -- 172.21.15.33:0/4167125706 wait complete. 2024-03-25T18:19:19.289 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.286+0000 7f4e3a681640 1 librados: shutdown 2024-03-25T18:19:19.299 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new 8a5817bc-3728-4133-879e-37557b37c520 6 2024-03-25T18:19:19.399 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.398+0000 7fae31172640 10 monclient: get_monmap_and_config 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.398+0000 7fae31172640 10 monclient: build_initial_monmap 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.398+0000 7fae31172640 10 monclient: monmap: 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:19.399293+0000 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:19.399293+0000 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.398+0000 7fae31172640 1 Processor -- start 2024-03-25T18:19:19.400 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.398+0000 7fae31172640 1 -- start start 2024-03-25T18:19:19.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.398+0000 7fae31172640 10 monclient: init 2024-03-25T18:19:19.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:19.401 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 10 monclient(hunting): picked mon.noname-a con 0x7fae2c0b4700 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 10 monclient(hunting): picked mon.noname-b con 0x7fae2c142a20 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 10 monclient(hunting): picked mon.noname-c con 0x7fae2c0a96b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c0a9d60 con 0x7fae2c0b4700 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c13ff10 con 0x7fae2c0a96b0 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c0aa1f0 con 0x7fae2c142a20 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae31172640 10 monclient(hunting): authenticate will time out at 7178.563921s 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae2ad9d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fae2c0b4700 0x7fae2c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.33:46452/0 (socket says 172.21.15.33:46452) 2024-03-25T18:19:19.402 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.399+0000 7fae2ad9d640 1 -- 172.21.15.33:0/2292127136 learned_addr learned my addr 172.21.15.33:0/2292127136 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:19.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.401+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095731403 0 0) 0x7fae2c0a9d60 con 0x7fae2c0b4700 2024-03-25T18:19:19.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.401+0000 7fae29d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.401+0000 7fae29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.401+0000 7fae29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.401+0000 7fae29d9b640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:19:19.403 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.401+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae18003520 con 0x7fae2c0b4700 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529869406 0 0) 0x7fae2c13ff10 con 0x7fae2c0a96b0 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c0a9d60 con 0x7fae2c0a96b0 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631873685 0 0) 0x7fae2c0aa1f0 con 0x7fae2c142a20 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:19:19.404 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.402+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c13ff10 con 0x7fae2c142a20 2024-03-25T18:19:19.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3250848120 0 0) 0x7fae2c0a9d60 con 0x7fae2c0a96b0 2024-03-25T18:19:19.406 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 >> v1:172.21.15.22:6790/0 conn(0x7fae2c142a20 legacy=0x7fae2c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 >> v1:172.21.15.22:6789/0 conn(0x7fae2c0b4700 legacy=0x7fae2c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 1 monclient: found mon.noname-c 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c0b4ae0 con 0x7fae2c0a96b0 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 10 monclient: _finish_auth 0 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fae240019f0 con 0x7fae2c0a96b0 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae29d9b640 10 monclient: dump: 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.407 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae31172640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:19:19.408 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.405+0000 7fae31172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae24001b50 con 0x7fae2c0a96b0 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae29d9b640 1 -- 172.21.15.33:0/2292127136 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fae24001f60 con 0x7fae2c0a96b0 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 4 set_mon_vals no callback set 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae29d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae29d9b640 10 monclient: dump: 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.409 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 10 monclient: shutdown 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 1 -- 172.21.15.33:0/2292127136 >> v1:172.21.15.33:6789/0 conn(0x7fae2c0a96b0 legacy=0x7fae2c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 1 -- 172.21.15.33:0/2292127136 shutdown_connections 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 1 -- 172.21.15.33:0/2292127136 >> 172.21.15.33:0/2292127136 conn(0x7fae2c0668e0 msgr2=0x7fae2c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 1 -- 172.21.15.33:0/2292127136 shutdown_connections 2024-03-25T18:19:19.410 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.407+0000 7fae31172640 1 -- 172.21.15.33:0/2292127136 wait complete. 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.408+0000 7fae31172640 10 monclient: build_initial_monmap 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.408+0000 7fae31172640 10 monclient: monmap: 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:19.412 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.408+0000 7fae31172640 1 librados: starting msgr at 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.408+0000 7fae31172640 1 librados: starting objecter 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 1 Processor -- start 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 1 -- start start 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 1 librados: setting wanted keys 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 1 librados: calling monclient init 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient: init 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient(hunting): picked mon.noname-c con 0x7fae2c0a96b0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient(hunting): picked mon.noname-a con 0x7fae2c0b4700 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient(hunting): picked mon.noname-b con 0x7fae2c142a20 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c138430 con 0x7fae2c0b4700 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c1d3fd0 con 0x7fae2c142a20 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2c1d4f30 con 0x7fae2c0a96b0 2024-03-25T18:19:19.413 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.409+0000 7fae31172640 10 monclient(hunting): authenticate will time out at 7178.574481s 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae2b59e640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fae2c142a20 0x7fae2c12ff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:60712/0 (socket says 172.21.15.33:60712) 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae2b59e640 1 -- 172.21.15.33:0/3812167694 learned_addr learned my addr 172.21.15.33:0/3812167694 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae237fe640 10 client.?.objecter ms_handle_connect 0x7fae2c142a20 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae237fe640 10 client.?.objecter ms_handle_connect 0x7fae2c0b4700 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae237fe640 10 client.?.objecter ms_handle_connect 0x7fae2c0a96b0 2024-03-25T18:19:19.414 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.410+0000 7fae237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520656065 0 0) 0x7fae2c1d3fd0 con 0x7fae2c142a20 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae0c004eb0 con 0x7fae2c142a20 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67120045 0 0) 0x7fae2c138430 con 0x7fae2c0b4700 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:19:19.415 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c1d3fd0 con 0x7fae2c0b4700 2024-03-25T18:19:19.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485140012 0 0) 0x7fae2c1d4f30 con 0x7fae2c0a96b0 2024-03-25T18:19:19.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:19.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:19.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:19.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:19:19.416 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.412+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae2c138430 con 0x7fae2c0a96b0 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 688370796 0 0) 0x7fae0c004eb0 con 0x7fae2c142a20 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae2c1d4f30 con 0x7fae2c142a20 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fae24001f60 con 0x7fae2c142a20 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 10 monclient(hunting): dump: 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.419 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745937537 0 0) 0x7fae2c1d3fd0 con 0x7fae2c0b4700 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae0c004eb0 con 0x7fae2c0b4700 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fae1c001810 con 0x7fae2c0b4700 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.417+0000 7fae237fe640 10 monclient(hunting): dump: 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.420 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.418+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387848116 0 0) 0x7fae2c138430 con 0x7fae2c0a96b0 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.418+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae2c1d3fd0 con 0x7fae2c0a96b0 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.418+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fae140017b0 con 0x7fae2c0a96b0 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.418+0000 7fae237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.418+0000 7fae237fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.418+0000 7fae237fe640 10 monclient(hunting): dump: 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.421 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.422 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.422 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1933734772 0 0) 0x7fae2c1d4f30 con 0x7fae2c142a20 2024-03-25T18:19:19.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 >> v1:172.21.15.22:6790/0 conn(0x7fae2c0a96b0 legacy=0x7fae2c137d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 >> v1:172.21.15.22:6789/0 conn(0x7fae2c0b4700 legacy=0x7fae2c1d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:19.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 1 monclient: found mon.b 2024-03-25T18:19:19.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:19.423 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c134380 con 0x7fae2c142a20 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 10 monclient: _renew_subs 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2c134630 con 0x7fae2c142a20 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 10 monclient: _renew_subs 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae2c137660 con 0x7fae2c142a20 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.420+0000 7fae31172640 1 librados: init done 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.421+0000 7fae31172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.421+0000 7fae31172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.421+0000 7fae31172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:19.424 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.421+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae2c1d4670 con 0x7fae2c142a20 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae24003c20 con 0x7fae2c142a20 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fae24003f70 con 0x7fae2c142a20 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 10 monclient: dump: 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:19.681 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fae24004520 con 0x7fae2c142a20 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae2959a640 4 set_mon_vals no callback set 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1715113353 0 0) 0x7fae24006c30 con 0x7fae2c142a20 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 10 client.4144.objecter ms_dispatch 0x7fae2c000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 3 client.4144.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 3 client.4144.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fae24018de0 con 0x7fae2c142a20 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:19.682 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.679+0000 7fae237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:19.764 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.763+0000 7fae31172640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8a5817bc-3728-4133-879e-37557b37c520", "id": 6}] 2024-03-25T18:19:19.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.763+0000 7fae31172640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8a5817bc-3728-4133-879e-37557b37c520", "id": 6}] 2024-03-25T18:19:19.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.763+0000 7fae31172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:19.765 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.763+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8a5817bc-3728-4133-879e-37557b37c520", "id": 6} v 0) v1 -- 0x7fae2c1d4160 con 0x7fae2c142a20 2024-03-25T18:19:19.772 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:19.771+0000 7fbf33610040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:19:19.796 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:19.795+0000 7f0efd02b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:19:19.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.887+0000 7fae237fe640 1 -- 172.21.15.33:0/3812167694 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8a5817bc-3728-4133-879e-37557b37c520", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2842882004 0 1113811849) 0x7fae2c1d4160 con 0x7fae2c142a20 2024-03-25T18:19:19.888 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.887+0000 7fae237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8a5817bc-3728-4133-879e-37557b37c520", "id": 6}] 2024-03-25T18:19:19.889 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.887+0000 7fae237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:19.889 INFO:teuthology.orchestra.run.smithi033.stdout:6 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 10 librados: watch_flush enter 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 10 librados: watch_flush exit 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 10 monclient: shutdown 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 >> v1:172.21.15.33:6789/0 conn(0x7fae2c142a20 legacy=0x7fae2c12ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 shutdown_connections 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 >> 172.21.15.33:0/3812167694 conn(0x7fae2c0668e0 msgr2=0x7fae2c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:19.890 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 shutdown_connections 2024-03-25T18:19:19.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 1 -- 172.21.15.33:0/3812167694 wait complete. 2024-03-25T18:19:19.891 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.888+0000 7fae31172640 1 librados: shutdown 2024-03-25T18:19:19.900 DEBUG:teuthology.orchestra.run.smithi033:> sudo ceph --cluster ceph osd new ddfa42ca-42f4-4851-a399-5d591d922824 7 2024-03-25T18:19:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.999+0000 7f8cc9055640 10 monclient: get_monmap_and_config 2024-03-25T18:19:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.999+0000 7f8cc9055640 10 monclient: build_initial_monmap 2024-03-25T18:19:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:19.999+0000 7f8cc9055640 10 monclient: monmap: 2024-03-25T18:19:20.000 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:19:20.000813+0000 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:19:20.000813+0000 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:20.001 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.002 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 1 Processor -- start 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 1 -- start start 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient: init 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc40b4700 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc4142a20 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc40a96b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40a9d60 con 0x7f8cc4142a20 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc413ff10 con 0x7f8cc40a96b0 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40aa1f0 con 0x7f8cc40b4700 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.000+0000 7f8cc9055640 10 monclient(hunting): authenticate will time out at 7179.165450s 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.001+0000 7f8cc2d9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f8cc40b4700 0x7f8cc4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.33:49760/0 (socket says 172.21.15.33:49760) 2024-03-25T18:19:20.003 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.001+0000 7f8cc2d9d640 1 -- 172.21.15.33:0/3731477410 learned_addr learned my addr 172.21.15.33:0/3731477410 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.003+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165270370 0 0) 0x7f8cc40aa1f0 con 0x7f8cc40b4700 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.003+0000 7f8cc1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.003+0000 7f8cc1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.003+0000 7f8cc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.003+0000 7f8cc1d9b640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.003+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cac003520 con 0x7f8cc40b4700 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453714428 0 0) 0x7f8cc40a9d60 con 0x7f8cc4142a20 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.005 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:19:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc40aa1f0 con 0x7f8cc4142a20 2024-03-25T18:19:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215961261 0 0) 0x7f8cc413ff10 con 0x7f8cc40a96b0 2024-03-25T18:19:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:19:20.006 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.004+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc40a9d60 con 0x7f8cc40a96b0 2024-03-25T18:19:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301243303 0 0) 0x7f8cac003520 con 0x7f8cc40b4700 2024-03-25T18:19:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 >> v1:172.21.15.33:6789/0 conn(0x7f8cc40a96b0 legacy=0x7f8cc413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 >> v1:172.21.15.22:6789/0 conn(0x7f8cc4142a20 legacy=0x7f8cc413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:20.008 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40b4ae0 con 0x7f8cc40b4700 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8cb4001690 con 0x7f8cc40b4700 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc1d9b640 10 monclient: dump: 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.009 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.010 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc9055640 5 monclient: authenticate success, global_id 4157 2024-03-25T18:19:20.010 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.007+0000 7f8cc9055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb40017f0 con 0x7f8cc40b4700 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc1d9b640 1 -- 172.21.15.33:0/3731477410 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8cb4001c00 con 0x7f8cc40b4700 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc1d9b640 10 monclient: dump: 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.011 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.012 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc9055640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:20.012 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.009+0000 7f8cc9055640 4 set_mon_vals no callback set 2024-03-25T18:19:20.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.010+0000 7f8cc9055640 10 monclient: shutdown 2024-03-25T18:19:20.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.010+0000 7f8cc9055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:20.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.010+0000 7f8cc9055640 1 -- 172.21.15.33:0/3731477410 >> v1:172.21.15.22:6790/0 conn(0x7f8cc40b4700 legacy=0x7f8cc4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.010+0000 7f8cc9055640 1 -- 172.21.15.33:0/3731477410 shutdown_connections 2024-03-25T18:19:20.013 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.010+0000 7f8cc9055640 1 -- 172.21.15.33:0/3731477410 >> 172.21.15.33:0/3731477410 conn(0x7f8cc40668e0 msgr2=0x7f8cc4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.010+0000 7f8cc9055640 1 -- 172.21.15.33:0/3731477410 shutdown_connections 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.010+0000 7f8cc9055640 1 -- 172.21.15.33:0/3731477410 wait complete. 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.011+0000 7f8cc9055640 10 monclient: build_initial_monmap 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.011+0000 7f8cc9055640 10 monclient: monmap: 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 0 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 0.000000 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:created 0.000000 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 1 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:20.014 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.012+0000 7f8cc9055640 1 librados: starting msgr at 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.012+0000 7f8cc9055640 1 librados: starting objecter 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.012+0000 7f8cc9055640 1 Processor -- start 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.012+0000 7f8cc9055640 1 -- start start 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.012+0000 7f8cc9055640 1 librados: setting wanted keys 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.012+0000 7f8cc9055640 1 librados: calling monclient init 2024-03-25T18:19:20.015 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.012+0000 7f8cc9055640 10 monclient: init 2024-03-25T18:19:20.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:20.017 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 10 monclient(hunting): picked mon.noname-b con 0x7f8cc40a96b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 10 monclient(hunting): picked mon.noname-a con 0x7f8cc40b4700 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 10 monclient(hunting): picked mon.noname-c con 0x7f8cc4142a20 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40adde0 con 0x7f8cc40b4700 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40ae0c0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc40ace10 con 0x7f8cc4142a20 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc9055640 10 monclient(hunting): authenticate will time out at 7179.178105s 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc2d9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f8cc40a96b0 0x7f8cc413a7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.33:60726/0 (socket says 172.21.15.33:60726) 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.013+0000 7f8cc2d9d640 1 -- 172.21.15.33:0/2914781378 learned_addr learned my addr 172.21.15.33:0/2914781378 (peer_addr_for_me v1:172.21.15.33:0/0) 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.014+0000 7f8cab7fe640 10 client.?.objecter ms_handle_connect 0x7f8cc4142a20 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.014+0000 7f8cab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:20.018 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.014+0000 7f8cab7fe640 10 client.?.objecter ms_handle_connect 0x7f8cc40b4700 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.014+0000 7f8cab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.014+0000 7f8cab7fe640 10 client.?.objecter ms_handle_connect 0x7f8cc40a96b0 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.014+0000 7f8cab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277452639 0 0) 0x7f8cc40ace10 con 0x7f8cc4142a20 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb0004eb0 con 0x7f8cc4142a20 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252375836 0 0) 0x7f8cc40ae0c0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.019 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc40ace10 con 0x7f8cc40a96b0 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176200579 0 0) 0x7f8cc40adde0 con 0x7f8cc40b4700 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:19:20.020 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.016+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cc40ae0c0 con 0x7f8cc40b4700 2024-03-25T18:19:20.022 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.020+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2967048471 0 0) 0x7f8cb0004eb0 con 0x7f8cc4142a20 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.020+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc40adde0 con 0x7f8cc4142a20 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.020+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8cbc001bd0 con 0x7f8cc4142a20 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.020+0000 7f8cab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.020+0000 7f8cab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.020+0000 7f8cab7fe640 10 monclient(hunting): dump: 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.023 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.020+0000 7f8cab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665653906 0 0) 0x7f8cc40ace10 con 0x7f8cc40a96b0 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cb0004eb0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8cb4001b40 con 0x7f8cc40a96b0 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 10 monclient(hunting): dump: 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.024 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 622082979 0 0) 0x7f8cc40ae0c0 con 0x7f8cc40b4700 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8cc40ace10 con 0x7f8cc40b4700 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8cb80018f0 con 0x7f8cc40b4700 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.021+0000 7f8cab7fe640 10 monclient(hunting): dump: 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:20.025 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.026 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.026 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3845895466 0 0) 0x7f8cb0004eb0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 >> v1:172.21.15.22:6790/0 conn(0x7f8cc4142a20 legacy=0x7f8cc40ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 >> v1:172.21.15.22:6789/0 conn(0x7f8cc40b4700 legacy=0x7f8cc40ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 1 monclient: found mon.b 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc40aba80 con 0x7f8cc40a96b0 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:20.027 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 10 monclient: _renew_subs 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc40ac900 con 0x7f8cc40a96b0 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 10 monclient: _renew_subs 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc41e3480 con 0x7f8cc40a96b0 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.023+0000 7f8cc9055640 1 librados: init done 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.024+0000 7f8cc9055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.024+0000 7f8cc9055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.024+0000 7f8cc9055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.028 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.024+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc40acfa0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.095 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:20.094+0000 7f0efd02b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:19:20.096 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:20.095+0000 7fbf33610040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:19:20.130 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cb4004a90 con 0x7f8cc40a96b0 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8cb4004da0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 10 monclient: dump: 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:epoch 1 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:election_strategy: 3 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.131 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8cb4005290 con 0x7f8cc40a96b0 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cc159a640 4 set_mon_vals no callback set 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 91309329 0 0) 0x7f8cb4006910 con 0x7f8cc40a96b0 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 10 client.4150.objecter ms_dispatch 0x7f8cc4000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 3 client.4150.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 3 client.4150.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8cb4018dc0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.132 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.129+0000 7f8cab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:20.149 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:20.148+0000 7f0efd02b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:19:20.149 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:20.148+0000 7fbf33610040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:19:20.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.202+0000 7f8cc9055640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ddfa42ca-42f4-4851-a399-5d591d922824", "id": 7}] 2024-03-25T18:19:20.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.202+0000 7f8cc9055640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ddfa42ca-42f4-4851-a399-5d591d922824", "id": 7}] 2024-03-25T18:19:20.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.202+0000 7f8cc9055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.204 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.202+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ddfa42ca-42f4-4851-a399-5d591d922824", "id": 7} v 0) v1 -- 0x7f8cc404f730 con 0x7f8cc40a96b0 2024-03-25T18:19:20.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.266+0000 7f8cab7fe640 1 -- 172.21.15.33:0/2914781378 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ddfa42ca-42f4-4851-a399-5d591d922824", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 285115927 0 1371665406) 0x7f8cb40063e0 con 0x7f8cc40a96b0 2024-03-25T18:19:20.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ddfa42ca-42f4-4851-a399-5d591d922824", "id": 7}] 2024-03-25T18:19:20.268 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:20.268 INFO:teuthology.orchestra.run.smithi033.stdout:7 2024-03-25T18:19:20.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 10 librados: watch_flush enter 2024-03-25T18:19:20.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 10 librados: watch_flush exit 2024-03-25T18:19:20.269 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 10 client.4150.objecter successfully canceled tick 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 10 monclient: shutdown 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 >> v1:172.21.15.33:6789/0 conn(0x7f8cc40a96b0 legacy=0x7f8cc413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 shutdown_connections 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.267+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 >> 172.21.15.33:0/2914781378 conn(0x7f8cc40668e0 msgr2=0x7f8cc40b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.268+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 shutdown_connections 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.268+0000 7f8cc9055640 1 -- 172.21.15.33:0/2914781378 wait complete. 2024-03-25T18:19:20.270 INFO:teuthology.orchestra.run.smithi033.stderr:2024-03-25T18:19:20.268+0000 7f8cc9055640 1 librados: shutdown 2024-03-25T18:19:20.272 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:20.271+0000 7f0efd02b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:19:20.279 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.280 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:19:20.280 DEBUG:teuthology.orchestra.run.smithi022:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:19:20.282 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:19:20.282 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.282 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:19:20.283 DEBUG:teuthology.orchestra.run.smithi022:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:19:20.283 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-25T18:19:20.282+0000 7fbf33610040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:19:20.285 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:19:20.285 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.285 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:19:20.285 DEBUG:teuthology.orchestra.run.smithi022:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:19:20.288 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:19:20.288 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.288 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:19:20.288 DEBUG:teuthology.orchestra.run.smithi022:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:19:20.290 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:19:20.291 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.291 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:19:20.291 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:19:20.293 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:19:20.293 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.293 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:19:20.293 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:19:20.296 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:19:20.296 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.296 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:19:20.296 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:19:20.298 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:19:20.298 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:19:20.298 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:19:20.299 DEBUG:teuthology.orchestra.run.smithi033:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:19:20.301 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:19:20.301 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.438+0000 7ff26a681640 10 monclient: get_monmap_and_config 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.438+0000 7ff26a681640 10 monclient: build_initial_monmap 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.438+0000 7ff26a681640 10 monclient: monmap: 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:19:20.440156+0000 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:19:20.440156+0000 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:20.440 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:20.441 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:20.441 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.439+0000 7ff26a681640 1 Processor -- start 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.439+0000 7ff26a681640 1 -- start start 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.439+0000 7ff26a681640 10 monclient: init 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 10 monclient(hunting): picked mon.noname-b con 0x7ff2640a9920 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 10 monclient(hunting): picked mon.noname-c con 0x7ff2640b45a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 10 monclient(hunting): picked mon.noname-a con 0x7ff264146450 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:20.442 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26406b100 con 0x7ff264146450 2024-03-25T18:19:20.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2640b2940 con 0x7ff2640b45a0 2024-03-25T18:19:20.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26413fec0 con 0x7ff2640a9920 2024-03-25T18:19:20.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:20.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.440+0000 7ff26a681640 10 monclient(hunting): authenticate will time out at 7159.664495s 2024-03-25T18:19:20.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.441+0000 7ff263fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff2640a9920 0x7ff26413f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54166/0 (socket says 172.21.15.22:54166) 2024-03-25T18:19:20.443 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.441+0000 7ff263fff640 1 -- 172.21.15.22:0/1657684943 learned_addr learned my addr 172.21.15.22:0/1657684943 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:20.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.580+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171480809 0 0) 0x7ff2640b2940 con 0x7ff2640b45a0 2024-03-25T18:19:20.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.580+0000 7ff262ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.580+0000 7ff262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.580+0000 7ff262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.580+0000 7ff262ffd640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:19:20.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.580+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c003520 con 0x7ff2640b45a0 2024-03-25T18:19:20.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413280516 0 0) 0x7ff24c003520 con 0x7ff2640b45a0 2024-03-25T18:19:20.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 >> v1:172.21.15.22:6790/0 conn(0x7ff2640a9920 legacy=0x7ff26413f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 >> v1:172.21.15.22:6789/0 conn(0x7ff264146450 legacy=0x7ff26414dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:20.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 1 monclient: found mon.noname-c 2024-03-25T18:19:20.591 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff264140050 con 0x7ff2640b45a0 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff258001690 con 0x7ff2640b45a0 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff262ffd640 10 monclient: dump: 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.592 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff26a681640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:19:20.593 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.590+0000 7ff26a681640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2580017f0 con 0x7ff2640b45a0 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff262ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff262ffd640 1 -- 172.21.15.22:0/1657684943 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff258001a60 con 0x7ff2640b45a0 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff262ffd640 10 monclient: dump: 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:20.617 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.618 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.618 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.618 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:20.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff26a681640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:20.618 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff26a681640 4 set_mon_vals no callback set 2024-03-25T18:19:20.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.616+0000 7ff26a681640 10 monclient: shutdown 2024-03-25T18:19:20.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.617+0000 7ff26a681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:20.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.617+0000 7ff26a681640 1 -- 172.21.15.22:0/1657684943 >> v1:172.21.15.33:6789/0 conn(0x7ff2640b45a0 legacy=0x7ff264145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.617+0000 7ff26a681640 1 -- 172.21.15.22:0/1657684943 shutdown_connections 2024-03-25T18:19:20.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.617+0000 7ff26a681640 1 -- 172.21.15.22:0/1657684943 >> 172.21.15.22:0/1657684943 conn(0x7ff2640b7f40 msgr2=0x7ff2640b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:20.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.617+0000 7ff26a681640 1 -- 172.21.15.22:0/1657684943 shutdown_connections 2024-03-25T18:19:20.619 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.617+0000 7ff26a681640 1 -- 172.21.15.22:0/1657684943 wait complete. 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.618+0000 7ff26a681640 10 monclient: build_initial_monmap 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.618+0000 7ff26a681640 10 monclient: monmap: 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:20.620 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:20.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.618+0000 7ff26a681640 1 librados: starting msgr at 2024-03-25T18:19:20.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.618+0000 7ff26a681640 1 librados: starting objecter 2024-03-25T18:19:20.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.618+0000 7ff26a681640 1 Processor -- start 2024-03-25T18:19:20.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.619+0000 7ff26a681640 1 -- start start 2024-03-25T18:19:20.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.619+0000 7ff26a681640 1 librados: setting wanted keys 2024-03-25T18:19:20.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.619+0000 7ff26a681640 1 librados: calling monclient init 2024-03-25T18:19:20.621 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.619+0000 7ff26a681640 10 monclient: init 2024-03-25T18:19:20.622 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.619+0000 7ff26a681640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.619+0000 7ff26a681640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 10 monclient(hunting): picked mon.noname-a con 0x7ff2640a9920 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 10 monclient(hunting): picked mon.noname-c con 0x7ff2640b45a0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 10 monclient(hunting): picked mon.noname-b con 0x7ff264146450 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2640ac2b0 con 0x7ff2640a9920 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2640ac590 con 0x7ff264146450 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2640ac870 con 0x7ff2640b45a0 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.620+0000 7ff26a681640 10 monclient(hunting): authenticate will time out at 7159.844053s 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.621+0000 7ff268c1e640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7ff264146450 0x7ff2641d8090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:40000/0 (socket says 172.21.15.22:40000) 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.621+0000 7ff268c1e640 1 -- 172.21.15.22:0/256174307 learned_addr learned my addr 172.21.15.22:0/256174307 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.621+0000 7ff260ff9640 10 client.?.objecter ms_handle_connect 0x7ff264146450 2024-03-25T18:19:20.623 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.621+0000 7ff260ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:20.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.624+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069327856 0 0) 0x7ff2640ac590 con 0x7ff264146450 2024-03-25T18:19:20.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.624+0000 7ff260ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:20.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.624+0000 7ff260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:20.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.624+0000 7ff260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:20.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.624+0000 7ff260ff9640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:19:20.626 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.624+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff250004eb0 con 0x7ff264146450 2024-03-25T18:19:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.632+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3772698990 0 0) 0x7ff250004eb0 con 0x7ff264146450 2024-03-25T18:19:20.633 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.632+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2640ac590 con 0x7ff264146450 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.632+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff25c001df0 con 0x7ff264146450 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.632+0000 7ff260ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.632+0000 7ff260ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.632+0000 7ff260ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.634 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.635 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:20.635 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.632+0000 7ff260ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2072512613 0 0) 0x7ff2640ac590 con 0x7ff264146450 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 >> v1:172.21.15.22:6790/0 conn(0x7ff2640b45a0 legacy=0x7ff2641d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 >> v1:172.21.15.22:6789/0 conn(0x7ff2640a9920 legacy=0x7ff264137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 1 monclient: found mon.b 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff264144940 con 0x7ff264146450 2024-03-25T18:19:20.641 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff260ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 10 monclient: _renew_subs 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640ab930 con 0x7ff264146450 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 10 monclient: _renew_subs 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff264137c00 con 0x7ff264146450 2024-03-25T18:19:20.642 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.639+0000 7ff26a681640 1 librados: init done 2024-03-25T18:19:20.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.641+0000 7ff26a681640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.641+0000 7ff26a681640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.641+0000 7ff26a681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.643 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.641+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff264144790 con 0x7ff264146450 2024-03-25T18:19:20.665 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff25c001f50 con 0x7ff264146450 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff260ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff25c003e60 con 0x7ff264146450 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff260ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff260ff9640 10 monclient: dump: 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:20.666 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:20.667 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:20.667 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:20.667 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:19:20.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff2627fc640 4 set_mon_vals no callback set 2024-03-25T18:19:20.667 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.664+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 968868239 0 0) 0x7ff25c004080 con 0x7ff264146450 2024-03-25T18:19:20.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.667+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7ff25c015a80 con 0x7ff264146450 2024-03-25T18:19:20.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.667+0000 7ff260ff9640 10 client.4156.objecter ms_dispatch 0x7ff264000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:20.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.667+0000 7ff260ff9640 3 client.4156.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:20.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.667+0000 7ff260ff9640 3 client.4156.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:20.668 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.667+0000 7ff260ff9640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T18:19:20.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.840+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff25c014f20 con 0x7ff264146450 2024-03-25T18:19:20.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.840+0000 7ff260ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:20.842 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.840+0000 7ff260ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:20.966 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.965+0000 7ff26a681640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:20.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.965+0000 7ff26a681640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:20.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.965+0000 7ff26a681640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:20.967 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:20.965+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff26404f730 con 0x7ff264146450 2024-03-25T18:19:21.217 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-25T18:19:21.217 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.215+0000 7ff260ff9640 1 -- 172.21.15.22:0/256174307 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7ff25c013520 con 0x7ff264146450 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.216+0000 7ff260ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.216+0000 7ff260ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 10 librados: watch_flush enter 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 10 librados: watch_flush exit 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 10 client.4156.objecter successfully canceled tick 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 10 monclient: shutdown 2024-03-25T18:19:21.219 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 >> v1:172.21.15.33:6789/0 conn(0x7ff264146450 legacy=0x7ff2641d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 shutdown_connections 2024-03-25T18:19:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 >> 172.21.15.22:0/256174307 conn(0x7ff2640b7f40 msgr2=0x7ff2640b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 shutdown_connections 2024-03-25T18:19:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 1 -- 172.21.15.22:0/256174307 wait complete. 2024-03-25T18:19:21.220 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-25T18:19:21.217+0000 7ff26a681640 1 librados: shutdown 2024-03-25T18:19:21.234 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:19:21.234 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:19:22.039 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:22.126 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:22.125+0000 7f0e9d59a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:19:22.175 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:19:22.174+0000 7f0e9d59a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:19:22.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.182+0000 7ff900c15640 10 monclient: get_monmap_and_config 2024-03-25T18:19:22.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.182+0000 7ff900c15640 10 monclient: build_initial_monmap 2024-03-25T18:19:22.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.182+0000 7ff900c15640 10 monclient: monmap: 2024-03-25T18:19:22.183 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:22.183 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:22.183 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:19:22.183461+0000 2024-03-25T18:19:22.184 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:19:22.183461+0000 2024-03-25T18:19:22.184 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:22.184 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:22.184 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:22.184 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:22.184 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:22.184 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:22.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.183+0000 7ff900c15640 1 Processor -- start 2024-03-25T18:19:22.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.183+0000 7ff900c15640 1 -- start start 2024-03-25T18:19:22.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.183+0000 7ff900c15640 10 monclient: init 2024-03-25T18:19:22.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:22.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:22.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 10 monclient(hunting): picked mon.noname-c con 0x7ff8fc0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:22.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 10 monclient(hunting): picked mon.noname-a con 0x7ff8fc0b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:22.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 10 monclient(hunting): picked mon.noname-b con 0x7ff8fc14be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:22.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc0af940 con 0x7ff8fc0b1730 2024-03-25T18:19:22.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc0b0a70 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc0b0d50 con 0x7ff8fc14be90 2024-03-25T18:19:22.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:22.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.184+0000 7ff900c15640 10 monclient(hunting): authenticate will time out at 7161.407921s 2024-03-25T18:19:22.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.245+0000 7ff8f9d9b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff8fc0b1730 0x7ff8fc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39834/0 (socket says 172.21.15.22:39834) 2024-03-25T18:19:22.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.245+0000 7ff8f9d9b640 1 -- 172.21.15.22:0/792610950 learned_addr learned my addr 172.21.15.22:0/792610950 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:22.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.259+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925223985 0 0) 0x7ff8fc0b0a70 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.259+0000 7ff8f959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.259+0000 7ff8f959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.259+0000 7ff8f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.259+0000 7ff8f959a640 10 monclient(hunting): my global_id is 4171 2024-03-25T18:19:22.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.259+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e4003520 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2294934721 0 0) 0x7ff8e4003520 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 >> v1:172.21.15.22:6790/0 conn(0x7ff8fc14be90 legacy=0x7ff8fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 >> v1:172.21.15.22:6789/0 conn(0x7ff8fc0b1730 legacy=0x7ff8fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 1 monclient: found mon.noname-c 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0b1030 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 10 monclient: _finish_auth 0 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8ec001690 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff8f959a640 10 monclient: dump: 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff900c15640 5 monclient: authenticate success, global_id 4171 2024-03-25T18:19:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.265+0000 7ff900c15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.267+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8ec0017f0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.267+0000 7ff8f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.267+0000 7ff900c15640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.267+0000 7ff900c15640 4 set_mon_vals no callback set 2024-03-25T18:19:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff8f959a640 1 -- 172.21.15.22:0/792610950 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8ec001c40 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff8f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff8f959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff8f959a640 10 monclient: dump: 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff900c15640 10 monclient: shutdown 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff900c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff900c15640 1 -- 172.21.15.22:0/792610950 >> v1:172.21.15.33:6789/0 conn(0x7ff8fc0b4d60 legacy=0x7ff8fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff900c15640 1 -- 172.21.15.22:0/792610950 shutdown_connections 2024-03-25T18:19:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff900c15640 1 -- 172.21.15.22:0/792610950 >> 172.21.15.22:0/792610950 conn(0x7ff8fc0617a0 msgr2=0x7ff8fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff900c15640 1 -- 172.21.15.22:0/792610950 shutdown_connections 2024-03-25T18:19:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.268+0000 7ff900c15640 1 -- 172.21.15.22:0/792610950 wait complete. 2024-03-25T18:19:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.269+0000 7ff900c15640 10 monclient: build_initial_monmap 2024-03-25T18:19:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.269+0000 7ff900c15640 10 monclient: monmap: 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.270+0000 7ff900c15640 1 librados: starting msgr at 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.270+0000 7ff900c15640 1 librados: starting objecter 2024-03-25T18:19:22.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.270+0000 7ff900c15640 1 Processor -- start 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.270+0000 7ff900c15640 1 -- start start 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.270+0000 7ff900c15640 1 librados: setting wanted keys 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.270+0000 7ff900c15640 1 librados: calling monclient init 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.270+0000 7ff900c15640 10 monclient: init 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.271+0000 7ff900c15640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.271+0000 7ff900c15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.271+0000 7ff900c15640 10 monclient(hunting): picked mon.noname-a con 0x7ff8fc0b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.272+0000 7ff900c15640 10 monclient(hunting): picked mon.noname-b con 0x7ff8fc0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.272+0000 7ff900c15640 10 monclient(hunting): picked mon.noname-c con 0x7ff8fc14be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.272+0000 7ff900c15640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc139090 con 0x7ff8fc0b1730 2024-03-25T18:19:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.272+0000 7ff900c15640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc1355c0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.272+0000 7ff900c15640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc135750 con 0x7ff8fc14be90 2024-03-25T18:19:22.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.272+0000 7ff900c15640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:22.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.272+0000 7ff900c15640 10 monclient(hunting): authenticate will time out at 7161.496175s 2024-03-25T18:19:22.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.274+0000 7ff8fad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff8fc14be90 0x7ff8fc1e72e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54186/0 (socket says 172.21.15.22:54186) 2024-03-25T18:19:22.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.274+0000 7ff8fad9d640 1 -- 172.21.15.22:0/1490371165 learned_addr learned my addr 172.21.15.22:0/1490371165 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:22.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.275+0000 7ff8eaffd640 10 client.?.objecter ms_handle_connect 0x7ff8fc14be90 2024-03-25T18:19:22.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.275+0000 7ff8eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:22.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.275+0000 7ff8eaffd640 10 client.?.objecter ms_handle_connect 0x7ff8fc0b4d60 2024-03-25T18:19:22.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.275+0000 7ff8eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.289+0000 7ff8eaffd640 10 client.?.objecter ms_handle_connect 0x7ff8fc0b1730 2024-03-25T18:19:22.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.289+0000 7ff8eaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:22.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.434+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051488448 0 0) 0x7ff8fc1355c0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.434+0000 7ff8eaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:22.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.434+0000 7ff8eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:22.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.434+0000 7ff8eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:22.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.434+0000 7ff8eaffd640 10 monclient(hunting): my global_id is 4174 2024-03-25T18:19:22.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.434+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc004eb0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.442+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2527665990 0 0) 0x7ff8dc004eb0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.442+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8fc1355c0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.442+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8f00018a0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.442+0000 7ff8eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.442+0000 7ff8eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:22.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.442+0000 7ff8eaffd640 10 monclient(hunting): dump: 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:22.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.442+0000 7ff8eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:22.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2409573512 0 0) 0x7ff8fc1355c0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 >> v1:172.21.15.22:6790/0 conn(0x7ff8fc14be90 legacy=0x7ff8fc1e72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 >> v1:172.21.15.22:6789/0 conn(0x7ff8fc0b1730 legacy=0x7ff8fc134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 1 monclient: found mon.b 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc142420 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff8eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 5 monclient: authenticate success, global_id 4174 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 10 monclient: _renew_subs 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc142670 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 10 client.4174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 10 monclient: _renew_subs 2024-03-25T18:19:22.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:22.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc1455b0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.448+0000 7ff900c15640 1 librados: init done 2024-03-25T18:19:22.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.449+0000 7ff900c15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:22.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.449+0000 7ff900c15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:22.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.449+0000 7ff900c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:22.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.449+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc139220 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.477+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8f0001d50 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.477+0000 7ff8eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:22.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.477+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8f00049c0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.477+0000 7ff8eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:22.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.477+0000 7ff8eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:22.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.477+0000 7ff8eaffd640 10 monclient: dump: 2024-03-25T18:19:22.478 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:22.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.477+0000 7ff8f8d99640 4 set_mon_vals no callback set 2024-03-25T18:19:22.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.478+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3311759649 0 0) 0x7ff8f0004be0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.479+0000 7ff8fa59c640 10 monclient: get_auth_request con 0x7ff8dc06da90 auth_method 0 2024-03-25T18:19:22.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.479+0000 7ff8eaffd640 10 client.4174.objecter ms_handle_connect 0x7ff8dc06da90 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.482+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7ff8f00547a0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.482+0000 7ff8eaffd640 10 client.4174.objecter ms_dispatch 0x7ff8fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.482+0000 7ff8eaffd640 3 client.4174.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.482+0000 7ff8eaffd640 3 client.4174.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.482+0000 7ff8eaffd640 20 client.4174.objecter dump_active .. 0 homeless 2024-03-25T18:19:22.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.921+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8f00500e0 con 0x7ff8fc0b4d60 2024-03-25T18:19:22.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.921+0000 7ff8eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:22.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:22.921+0000 7ff8eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:23.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.291+0000 7ff900c15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.291+0000 7ff900c15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.291+0000 7ff900c15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:23.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.291+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8fc0627b0 con 0x7ff8fc0b4d60 2024-03-25T18:19:23.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.316+0000 7ff8eaffd640 1 -- 172.21.15.22:0/1490371165 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7ff8f00521e0 con 0x7ff8fc0b4d60 2024-03-25T18:19:23.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.316+0000 7ff8eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.316+0000 7ff8eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:23.317 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:19:23.317 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.318+0000 7ff900c15640 10 librados: watch_flush enter 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 10 librados: watch_flush exit 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 20 client.4174.objecter shutdown clearing up homeless session... 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 10 client.4174.objecter successfully canceled tick 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 >> v1:172.21.15.33:6800/215350958 conn(0x7ff8dc06da90 legacy=0x7ff8dc06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 10 monclient: shutdown 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 >> v1:172.21.15.33:6789/0 conn(0x7ff8fc0b4d60 legacy=0x7ff8fc1dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 shutdown_connections 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 >> 172.21.15.22:0/1490371165 conn(0x7ff8fc0617a0 msgr2=0x7ff8fc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 shutdown_connections 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 1 -- 172.21.15.22:0/1490371165 wait complete. 2024-03-25T18:19:23.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:23.319+0000 7ff900c15640 1 librados: shutdown 2024-03-25T18:19:23.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:27.336 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:06.979 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.399 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:07.042 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.440 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:07.083 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.450 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:07.085 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.502 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:07.145 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.515 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:07.150 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.571 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:07.211 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.588 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:07.222 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.630 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:07.271 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.651 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:07.285 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.714 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:07.358 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.787 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:07.431 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.883 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:07.511 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.943 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:07.571 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:27.944 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-03-25T18:19:27.937+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:27.959 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:07.586 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.022 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:07.649 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.063 INFO:tasks.ceph.osd.7.smithi033.stderr:2024-03-25T18:19:28.054+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:28.093 INFO:tasks.ceph.osd.5.smithi033.stderr:2024-03-25T18:19:28.087+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:28.206 INFO:tasks.ceph.osd.0.smithi022.stderr:2024-03-25T18:19:28.199+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:28.222 INFO:tasks.ceph.osd.6.smithi033.stderr:2024-03-25T18:19:28.214+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:28.335 INFO:tasks.ceph.osd.2.smithi022.stderr:2024-03-25T18:19:28.325+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:28.424 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:08.068 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.492 INFO:tasks.ceph.osd.3.smithi022.stderr:2024-03-25T18:19:28.485+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:28.519 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:08.154 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.576 INFO:tasks.ceph.osd.1.smithi022.stderr:2024-03-25T18:19:28.568+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:19:28.583 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:08.225 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.688 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:08.322 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.693 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:08.333 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.793 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:08.436 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.799 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:08.443 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.892 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:08.527 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.965 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:08.607 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:28.980 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:08.608 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.056 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:08.683 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.058 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:08.692 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.063 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:08.703 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.149 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:08.792 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.165 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:08.809 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.250 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:08.885 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.326 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:08.968 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.348 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:08.976 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.373 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:09.007 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.379 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:09.020 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.420 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:09.047 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.438 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:09.081 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.533 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:09.178 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.539 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:09.174 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.615 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:09.257 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.642 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:09.276 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.646 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:09.287 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.675 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:09.303 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.741 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:09.384 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.784 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:09.411 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.825 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:09.469 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.829 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:09.464 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.907 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:09.541 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.908 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:09.550 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.933 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:09.574 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:29.943 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:09.572 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.030 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:09.673 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.073 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:09.700 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.098 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:09.733 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.116 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:09.760 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.140 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:30.180 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:09.814 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.198 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:09.840 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.202 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:09.843 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.216 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:09.845 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.230+0000 7fc981d46640 10 monclient: get_monmap_and_config 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.230+0000 7fc981d46640 10 monclient: build_initial_monmap 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.230+0000 7fc981d46640 10 monclient: monmap: 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:19:30.231443+0000 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:19:30.231443+0000 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:30.231 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.230+0000 7fc981d46640 1 Processor -- start 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.230+0000 7fc981d46640 1 -- start start 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.230+0000 7fc981d46640 10 monclient: init 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 10 monclient(hunting): picked mon.noname-b con 0x7fc97c0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 10 monclient(hunting): picked mon.noname-c con 0x7fc97c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 10 monclient(hunting): picked mon.noname-a con 0x7fc97c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c142520 con 0x7fc97c1430c0 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c0b28a0 con 0x7fc97c13f710 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c0b2a30 con 0x7fc97c0a98f0 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.231+0000 7fc981d46640 10 monclient(hunting): authenticate will time out at 7169.455035s 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.232+0000 7fc97b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc97c0a98f0 0x7fc97c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41278/0 (socket says 172.21.15.22:41278) 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.232+0000 7fc97b7fe640 1 -- 172.21.15.22:0/2114697518 learned_addr learned my addr 172.21.15.22:0/2114697518 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:30.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.232+0000 7fc97bfff640 1 --1- 172.21.15.22:0/2114697518 >> v1:172.21.15.22:6789/0 conn(0x7fc97c1430c0 0x7fc97c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:54778/0 (socket says 172.21.15.22:54778) 2024-03-25T18:19:30.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061537632 0 0) 0x7fc97c0b2a30 con 0x7fc97c0a98f0 2024-03-25T18:19:30.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc964003520 con 0x7fc97c0a98f0 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797964395 0 0) 0x7fc97c142520 con 0x7fc97c1430c0 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c0b2a30 con 0x7fc97c1430c0 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473758658 0 0) 0x7fc97c0b28a0 con 0x7fc97c13f710 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:19:30.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.234+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c142520 con 0x7fc97c13f710 2024-03-25T18:19:30.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022368125 0 0) 0x7fc964003520 con 0x7fc97c0a98f0 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 >> v1:172.21.15.33:6789/0 conn(0x7fc97c13f710 legacy=0x7fc97c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 >> v1:172.21.15.22:6789/0 conn(0x7fc97c1430c0 legacy=0x7fc97c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c0b44c0 con 0x7fc97c0a98f0 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc96c001690 con 0x7fc97c0a98f0 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc981d46640 5 monclient: authenticate success, global_id 4205 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc981d46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:30.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc97a7fc640 10 monclient: dump: 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.237+0000 7fc981d46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:30.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc96c0017f0 con 0x7fc97c0a98f0 2024-03-25T18:19:30.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc97a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc97a7fc640 1 -- 172.21.15.22:0/2114697518 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc96c001c00 con 0x7fc97c0a98f0 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc981d46640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc981d46640 4 set_mon_vals no callback set 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc97a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc97a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.239+0000 7fc97a7fc640 10 monclient: dump: 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:30.242 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 10 monclient: shutdown 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 1 -- 172.21.15.22:0/2114697518 >> v1:172.21.15.22:6790/0 conn(0x7fc97c0a98f0 legacy=0x7fc97c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 1 -- 172.21.15.22:0/2114697518 shutdown_connections 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 1 -- 172.21.15.22:0/2114697518 >> 172.21.15.22:0/2114697518 conn(0x7fc97c0b7ea0 msgr2=0x7fc97c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 1 -- 172.21.15.22:0/2114697518 shutdown_connections 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 1 -- 172.21.15.22:0/2114697518 wait complete. 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 10 monclient: build_initial_monmap 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.240+0000 7fc981d46640 10 monclient: monmap: 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:30.243 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:19:30.244 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:30.244 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:30.244 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:30.244 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 librados: starting msgr at 2024-03-25T18:19:30.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 librados: starting objecter 2024-03-25T18:19:30.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 Processor -- start 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 -- start start 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 librados: setting wanted keys 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 librados: calling monclient init 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient: init 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient(hunting): picked mon.noname-b con 0x7fc97c0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient(hunting): picked mon.noname-a con 0x7fc97c13f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient(hunting): picked mon.noname-c con 0x7fc97c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c138b20 con 0x7fc97c13f710 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c137a80 con 0x7fc97c0a98f0 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97c139880 con 0x7fc97c1430c0 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.241+0000 7fc981d46640 10 monclient(hunting): authenticate will time out at 7169.465445s 2024-03-25T18:19:30.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc97bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc97c1430c0 0x7fc97c1e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41290/0 (socket says 172.21.15.22:41290) 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc97bfff640 1 -- 172.21.15.22:0/361828825 learned_addr learned my addr 172.21.15.22:0/361828825 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc963fff640 10 client.?.objecter ms_handle_connect 0x7fc97c1430c0 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc963fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc963fff640 10 client.?.objecter ms_handle_connect 0x7fc97c13f710 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc963fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc963fff640 10 client.?.objecter ms_handle_connect 0x7fc97c0a98f0 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.242+0000 7fc963fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152561715 0 0) 0x7fc97c137a80 con 0x7fc97c0a98f0 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc968004eb0 con 0x7fc97c0a98f0 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917098693 0 0) 0x7fc97c139880 con 0x7fc97c1430c0 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c137a80 con 0x7fc97c1430c0 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921915408 0 0) 0x7fc97c138b20 con 0x7fc97c13f710 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:19:30.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.244+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c139880 con 0x7fc97c13f710 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1970126550 0 0) 0x7fc968004eb0 con 0x7fc97c0a98f0 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97c138b20 con 0x7fc97c0a98f0 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4106236027 0 0) 0x7fc97c137a80 con 0x7fc97c1430c0 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc968004eb0 con 0x7fc97c1430c0 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc974001c50 con 0x7fc97c1430c0 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): dump: 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.251 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2036259274 0 0) 0x7fc97c139880 con 0x7fc97c13f710 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97c137a80 con 0x7fc97c13f710 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc96c001b80 con 0x7fc97c0a98f0 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): dump: 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.252 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc9700017f0 con 0x7fc97c13f710 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.249+0000 7fc963fff640 10 monclient(hunting): dump: 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:30.253 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1078842349 0 0) 0x7fc968004eb0 con 0x7fc97c1430c0 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 >> v1:172.21.15.33:6789/0 conn(0x7fc97c0a98f0 legacy=0x7fc97c137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 >> v1:172.21.15.22:6789/0 conn(0x7fc97c13f710 legacy=0x7fc97c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 1 monclient: found mon.c 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c139a10 con 0x7fc97c1430c0 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc963fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc981d46640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:19:30.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc981d46640 10 monclient: _renew_subs 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc981d46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.252+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc97c1dab00 con 0x7fc97c1430c0 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.253+0000 7fc981d46640 10 client.4208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.253+0000 7fc981d46640 10 monclient: _renew_subs 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.253+0000 7fc981d46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.253+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97c13b4f0 con 0x7fc97c1430c0 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.253+0000 7fc981d46640 1 librados: init done 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.254+0000 7fc981d46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.254+0000 7fc981d46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.254+0000 7fc981d46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:30.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.254+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc97c074cc0 con 0x7fc97c1430c0 2024-03-25T18:19:30.319 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:09.962 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.365 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:10.000 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.368 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:09.995 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.408 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:10.053 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.453 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:10.087 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.467 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:10.107 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.470+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9740020c0 con 0x7fc97c1430c0 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.470+0000 7fc963fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.470+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc974004d00 con 0x7fc97c1430c0 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.470+0000 7fc963fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.470+0000 7fc963fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.470+0000 7fc963fff640 10 monclient: dump: 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:30.471 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:30.472 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:30.472 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:30.472 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:30.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.470+0000 7fc979ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:30.485 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:10.114 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.490 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:10.132 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.608 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:10.251 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.620+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fc97401acd0 con 0x7fc97c1430c0 2024-03-25T18:19:30.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.621+0000 7fc97b7fe640 10 monclient: get_auth_request con 0x7fc96806e060 auth_method 0 2024-03-25T18:19:30.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.621+0000 7fc963fff640 10 client.4208.objecter ms_handle_connect 0x7fc96806e060 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fc974054a50 con 0x7fc97c1430c0 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 10 client.4208.objecter ms_dispatch 0x7fc97c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 3 client.4208.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 3 client.4208.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 20 client.4208.objecter dump_active .. 0 homeless 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc974050390 con 0x7fc97c1430c0 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.626+0000 7fc963fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:30.659 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:10.294 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.679 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:10.306 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.724 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:10.358 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.725 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:10.370 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.734 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:10.375 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.752 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:10.381 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.789 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:10.431 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.813+0000 7fc981d46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:30.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.813+0000 7fc981d46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:30.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.813+0000 7fc981d46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:30.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.813+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc97c0b3e80 con 0x7fc97c1430c0 2024-03-25T18:19:30.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.880+0000 7fc963fff640 1 -- 172.21.15.22:0/361828825 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fc974052490 con 0x7fc97c1430c0 2024-03-25T18:19:30.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.880+0000 7fc963fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:30.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.880+0000 7fc963fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:30.882 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:19:30.882 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:30.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 10 librados: watch_flush enter 2024-03-25T18:19:30.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 10 librados: watch_flush exit 2024-03-25T18:19:30.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 20 client.4208.objecter shutdown clearing up homeless session... 2024-03-25T18:19:30.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 10 client.4208.objecter successfully canceled tick 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 >> v1:172.21.15.33:6800/215350958 conn(0x7fc96806e060 legacy=0x7fc968070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 10 monclient: shutdown 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.882+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 >> v1:172.21.15.22:6790/0 conn(0x7fc97c1430c0 legacy=0x7fc97c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.883+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 shutdown_connections 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.883+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 >> 172.21.15.22:0/361828825 conn(0x7fc97c0b7ea0 msgr2=0x7fc97c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.883+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 shutdown_connections 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.883+0000 7fc981d46640 1 -- 172.21.15.22:0/361828825 wait complete. 2024-03-25T18:19:30.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:30.883+0000 7fc981d46640 1 librados: shutdown 2024-03-25T18:19:30.894 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:30.897 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:10.540 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.949 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:10.584 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.981 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:10.608 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.993 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:10.627 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:30.999 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:10.640 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.017 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:10.661 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.029 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:10.658 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.078 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:10.720 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.278 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:10.905 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.301 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:10.930 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.314 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:10.958 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.433 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:11.077 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.474 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:11.108 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.489 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:11.132 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.495 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:11.136 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.502 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:11.137 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.529 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:11.163 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.551 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:11.192 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.558 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:11.194 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.578 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:11.205 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.638 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:11.280 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.694 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:11.337 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.787 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:11.415 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.842 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:11.471 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.844 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:11.489 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:31.900 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:11.545 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:32.173 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:11.800 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:32.229 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:11.856 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:36.971 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:16.605 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:37.206 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:16.846 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:37.259 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:16.902 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:37.402 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:17.030 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:37.472 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:17.116 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:37.499 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:17.134 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:37.699 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:37.719 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:17.361 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.808+0000 7efda650e640 10 monclient: get_monmap_and_config 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.808+0000 7efda650e640 10 monclient: build_initial_monmap 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.808+0000 7efda650e640 10 monclient: monmap: 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:19:37.810186+0000 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:19:37.810186+0000 2024-03-25T18:19:37.810 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:37.811 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:37.811 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:37.811 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:37.811 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:37.811 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.809+0000 7efda650e640 1 Processor -- start 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.809+0000 7efda650e640 1 -- start start 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.809+0000 7efda650e640 10 monclient: init 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.810+0000 7efda650e640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.810+0000 7efda650e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.810+0000 7efda650e640 10 monclient(hunting): picked mon.noname-c con 0x7efda00b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.810+0000 7efda650e640 10 monclient(hunting): picked mon.noname-b con 0x7efda00b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.810+0000 7efda650e640 10 monclient(hunting): picked mon.noname-a con 0x7efda014be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.811+0000 7efda650e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00af940 con 0x7efda014be90 2024-03-25T18:19:37.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.811+0000 7efda650e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00b0a70 con 0x7efda00b4d60 2024-03-25T18:19:37.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.811+0000 7efda650e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda00b0d50 con 0x7efda00b1730 2024-03-25T18:19:37.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.811+0000 7efda650e640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:37.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.811+0000 7efda650e640 10 monclient(hunting): authenticate will time out at 7177.034915s 2024-03-25T18:19:37.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.811+0000 7efd9ffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7efda00b4d60 0x7efda013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:40604/0 (socket says 172.21.15.22:40604) 2024-03-25T18:19:37.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.811+0000 7efd9ffff640 1 -- 172.21.15.22:0/2085411446 learned_addr learned my addr 172.21.15.22:0/2085411446 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:37.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.814+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807699789 0 0) 0x7efda00b0a70 con 0x7efda00b4d60 2024-03-25T18:19:37.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.814+0000 7efd9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.814+0000 7efd9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.814+0000 7efd9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.814+0000 7efd9effd640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:19:37.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.814+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd88003520 con 0x7efda00b4d60 2024-03-25T18:19:37.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814063350 0 0) 0x7efda00b0d50 con 0x7efda00b1730 2024-03-25T18:19:37.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:19:37.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda00b0a70 con 0x7efda00b1730 2024-03-25T18:19:37.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381991534 0 0) 0x7efda00af940 con 0x7efda014be90 2024-03-25T18:19:37.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:19:37.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.815+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda00b0d50 con 0x7efda014be90 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1054972289 0 0) 0x7efda00b0a70 con 0x7efda00b1730 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 >> v1:172.21.15.33:6789/0 conn(0x7efda00b4d60 legacy=0x7efda013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 >> v1:172.21.15.22:6789/0 conn(0x7efda014be90 legacy=0x7efda014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 1 monclient: found mon.noname-b 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda00b1030 con 0x7efda00b1730 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:37.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efd9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efda650e640 5 monclient: authenticate success, global_id 4211 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.820+0000 7efda650e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.821+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efd94001630 con 0x7efda00b1730 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.821+0000 7efd9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.821+0000 7efd9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.821+0000 7efd9effd640 10 monclient: dump: 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:37.824 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:37.825 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:37.825 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.821+0000 7efda650e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.823+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd94001880 con 0x7efda00b1730 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.823+0000 7efd9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.824+0000 7efd9effd640 1 -- 172.21.15.22:0/2085411446 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efd94001c10 con 0x7efda00b1730 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.824+0000 7efda650e640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.824+0000 7efda650e640 4 set_mon_vals no callback set 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efd9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efd9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efda650e640 10 monclient: shutdown 2024-03-25T18:19:37.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efd9effd640 10 monclient: dump: 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efda650e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:37.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efda650e640 1 -- 172.21.15.22:0/2085411446 >> v1:172.21.15.22:6790/0 conn(0x7efda00b1730 legacy=0x7efda00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efda650e640 1 -- 172.21.15.22:0/2085411446 shutdown_connections 2024-03-25T18:19:37.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.825+0000 7efda650e640 1 -- 172.21.15.22:0/2085411446 >> 172.21.15.22:0/2085411446 conn(0x7efda00617a0 msgr2=0x7efda0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:37.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.826+0000 7efda650e640 1 -- 172.21.15.22:0/2085411446 shutdown_connections 2024-03-25T18:19:37.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.826+0000 7efda650e640 1 -- 172.21.15.22:0/2085411446 wait complete. 2024-03-25T18:19:37.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.826+0000 7efda650e640 10 monclient: build_initial_monmap 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.826+0000 7efda650e640 10 monclient: monmap: 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.827+0000 7efda650e640 1 librados: starting msgr at 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.827+0000 7efda650e640 1 librados: starting objecter 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.827+0000 7efda650e640 1 Processor -- start 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.827+0000 7efda650e640 1 -- start start 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.827+0000 7efda650e640 1 librados: setting wanted keys 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.827+0000 7efda650e640 1 librados: calling monclient init 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.827+0000 7efda650e640 10 monclient: init 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.828+0000 7efda650e640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:37.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.828+0000 7efda650e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:37.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.828+0000 7efda650e640 10 monclient(hunting): picked mon.noname-a con 0x7efda00b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:37.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.828+0000 7efda650e640 10 monclient(hunting): picked mon.noname-b con 0x7efda00b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:37.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efda650e640 10 monclient(hunting): picked mon.noname-c con 0x7efda014be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:37.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efda650e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda01418e0 con 0x7efda00b1730 2024-03-25T18:19:37.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efda650e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda01dae10 con 0x7efda00b4d60 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efda650e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda0141b50 con 0x7efda014be90 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efda650e640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efda650e640 10 monclient(hunting): authenticate will time out at 7177.052878s 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efd9ffff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7efda00b1730 0x7efda01405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58668/0 (socket says 172.21.15.22:58668) 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efd9ffff640 1 -- 172.21.15.22:0/2164635421 learned_addr learned my addr 172.21.15.22:0/2164635421 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efd9cff9640 10 client.?.objecter ms_handle_connect 0x7efda00b4d60 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efd9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efd9cff9640 10 client.?.objecter ms_handle_connect 0x7efda014be90 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.829+0000 7efd9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.830+0000 7efd9cff9640 10 client.?.objecter ms_handle_connect 0x7efda00b1730 2024-03-25T18:19:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.830+0000 7efd9cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.832+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125341873 0 0) 0x7efda01dae10 con 0x7efda00b4d60 2024-03-25T18:19:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.832+0000 7efd9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.832+0000 7efd9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.832+0000 7efd9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.832+0000 7efd9cff9640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.832+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c004eb0 con 0x7efda00b4d60 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423763522 0 0) 0x7efda0141b50 con 0x7efda014be90 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda01dae10 con 0x7efda014be90 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112334245 0 0) 0x7efda01418e0 con 0x7efda00b1730 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:19:37.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.833+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda0141b50 con 0x7efda00b1730 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1522330757 0 0) 0x7efd8c004eb0 con 0x7efda00b4d60 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda01418e0 con 0x7efda00b4d60 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efd94001b90 con 0x7efda00b4d60 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:37.839 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3476476242 0 0) 0x7efda0141b50 con 0x7efda00b1730 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd8c004eb0 con 0x7efda00b1730 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efd90001950 con 0x7efda00b1730 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:37.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.837+0000 7efd9cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:37.841 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.840+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 355573384 0 0) 0x7efda01dae10 con 0x7efda014be90 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.840+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda0141b50 con 0x7efda014be90 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.840+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efd98001b70 con 0x7efda014be90 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.840+0000 7efd9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.840+0000 7efd9cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.840+0000 7efd9cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:37.842 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.843 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.843 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.843 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:37.843 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:37.843 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:37.843 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:37.843 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1046962474 0 0) 0x7efda01418e0 con 0x7efda00b4d60 2024-03-25T18:19:37.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 >> v1:172.21.15.22:6790/0 conn(0x7efda014be90 legacy=0x7efda01428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 >> v1:172.21.15.22:6789/0 conn(0x7efda00b1730 legacy=0x7efda01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:37.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 1 monclient: found mon.b 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0142330 con 0x7efda00b4d60 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efd9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 5 monclient: authenticate success, global_id 4210 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 10 monclient: _renew_subs 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda01dba00 con 0x7efda00b4d60 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:37.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 10 monclient: _renew_subs 2024-03-25T18:19:37.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:37.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efda0138080 con 0x7efda00b4d60 2024-03-25T18:19:37.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.841+0000 7efda650e640 1 librados: init done 2024-03-25T18:19:37.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.843+0000 7efda650e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:37.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.843+0000 7efda650e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:37.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.843+0000 7efda650e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:37.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.843+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda0137140 con 0x7efda00b4d60 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.871+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd940038a0 con 0x7efda00b4d60 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.871+0000 7efd9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.871+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efd94003bb0 con 0x7efda00b4d60 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.871+0000 7efd9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.871+0000 7efd9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.871+0000 7efd9cff9640 10 monclient: dump: 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:37.872 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:37.873 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:37.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.871+0000 7efd9e7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:37.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.872+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7efd94019b80 con 0x7efda00b4d60 2024-03-25T18:19:37.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.873+0000 7efd9ffff640 10 monclient: get_auth_request con 0x7efd8c06e0d0 auth_method 0 2024-03-25T18:19:37.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.873+0000 7efd9cff9640 10 client.4210.objecter ms_handle_connect 0x7efd8c06e0d0 2024-03-25T18:19:37.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.875+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7efd940536e0 con 0x7efda00b4d60 2024-03-25T18:19:37.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.875+0000 7efd9cff9640 10 client.4210.objecter ms_dispatch 0x7efda0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:37.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.875+0000 7efd9cff9640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:37.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.875+0000 7efd9cff9640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:37.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:37.875+0000 7efd9cff9640 20 client.4210.objecter dump_active .. 0 homeless 2024-03-25T18:19:37.920 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:17.547 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.216+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd9404f020 con 0x7efda00b4d60 2024-03-25T18:19:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.216+0000 7efd9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:38.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.216+0000 7efd9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:38.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.433+0000 7efda650e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:38.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.433+0000 7efda650e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:38.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.433+0000 7efda650e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:19:38.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.433+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efda01424c0 con 0x7efda00b4d60 2024-03-25T18:19:38.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.451+0000 7efd9cff9640 1 -- 172.21.15.22:0/2164635421 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7efd94051120 con 0x7efda00b4d60 2024-03-25T18:19:38.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.451+0000 7efd9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:38.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.451+0000 7efd9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:38.452 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:19:38.452 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:38.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 10 librados: watch_flush enter 2024-03-25T18:19:38.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 10 librados: watch_flush exit 2024-03-25T18:19:38.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 20 client.4210.objecter shutdown clearing up homeless session... 2024-03-25T18:19:38.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 10 client.4210.objecter successfully canceled tick 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 >> v1:172.21.15.33:6800/215350958 conn(0x7efd8c06e0d0 legacy=0x7efd8c070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 10 monclient: shutdown 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 >> v1:172.21.15.33:6789/0 conn(0x7efda00b4d60 legacy=0x7efda01db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 shutdown_connections 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.453+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 >> 172.21.15.22:0/2164635421 conn(0x7efda00617a0 msgr2=0x7efda013c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.454+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 shutdown_connections 2024-03-25T18:19:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.454+0000 7efda650e640 1 -- 172.21.15.22:0/2164635421 wait complete. 2024-03-25T18:19:38.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:38.454+0000 7efda650e640 1 librados: shutdown 2024-03-25T18:19:38.465 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:42.908 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:22.541 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:42.911 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:22.545 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.003 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:22.644 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.007 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:22.648 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.082 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:22.725 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.088 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:22.731 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.127 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:22.769 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.131 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:22.773 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.249 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:22.894 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.253 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:22.898 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.254 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:22.882 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.257 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:22.886 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.459 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:23.095 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.463 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:23.099 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.554 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:23.181 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:43.560 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:23.187 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:45.271 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.375+0000 7fb6a6872640 10 monclient: get_monmap_and_config 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.375+0000 7fb6a6872640 10 monclient: build_initial_monmap 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.375+0000 7fb6a6872640 10 monclient: monmap: 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:19:45.376376+0000 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:19:45.376376+0000 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:45.376 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:45.377 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:45.377 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:45.377 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:45.377 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.375+0000 7fb6a6872640 1 Processor -- start 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.375+0000 7fb6a6872640 1 -- start start 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.375+0000 7fb6a6872640 10 monclient: init 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a00b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a00b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a014be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a00af940 con 0x7fb6a00b1730 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a00b0a70 con 0x7fb6a00b4d60 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a00b0d50 con 0x7fb6a014be90 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.376+0000 7fb6a6872640 10 monclient(hunting): authenticate will time out at 7184.600282s 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.377+0000 7fb69f7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb6a00b1730 0x7fb6a00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46460/0 (socket says 172.21.15.22:46460) 2024-03-25T18:19:45.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.377+0000 7fb69f7fe640 1 -- 172.21.15.22:0/1032117147 learned_addr learned my addr 172.21.15.22:0/1032117147 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:45.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.379+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693491626 0 0) 0x7fb6a00af940 con 0x7fb6a00b1730 2024-03-25T18:19:45.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.379+0000 7fb69effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.379+0000 7fb69effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.379+0000 7fb69effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.379+0000 7fb69effd640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:19:45.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.379+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb688003520 con 0x7fb6a00b1730 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236213496 0 0) 0x7fb6a00b0d50 con 0x7fb6a014be90 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb688003840 con 0x7fb6a014be90 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276024979 0 0) 0x7fb6a00b0a70 con 0x7fb6a00b4d60 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:19:45.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.380+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a00b0d50 con 0x7fb6a00b4d60 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4283212397 0 0) 0x7fb688003520 con 0x7fb6a00b1730 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 >> v1:172.21.15.22:6790/0 conn(0x7fb6a014be90 legacy=0x7fb6a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 >> v1:172.21.15.33:6789/0 conn(0x7fb6a00b4d60 legacy=0x7fb6a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 1 monclient: found mon.noname-a 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a00b1030 con 0x7fb6a00b1730 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 10 monclient: _finish_auth 0 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:45.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb6a6872640 5 monclient: authenticate success, global_id 4215 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb6a6872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb694001670 con 0x7fb6a00b1730 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb69effd640 10 monclient: dump: 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:45.387 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.383+0000 7fb6a6872640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb694001850 con 0x7fb6a00b1730 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb69effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb69effd640 1 -- 172.21.15.22:0/1032117147 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb694001670 con 0x7fb6a00b1730 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb69effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb69effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb69effd640 10 monclient: dump: 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:45.390 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb6a6872640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.385+0000 7fb6a6872640 4 set_mon_vals no callback set 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.386+0000 7fb6a6872640 10 monclient: shutdown 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.386+0000 7fb6a6872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.386+0000 7fb6a6872640 1 -- 172.21.15.22:0/1032117147 >> v1:172.21.15.22:6789/0 conn(0x7fb6a00b1730 legacy=0x7fb6a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.386+0000 7fb6a6872640 1 -- 172.21.15.22:0/1032117147 shutdown_connections 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.386+0000 7fb6a6872640 1 -- 172.21.15.22:0/1032117147 >> 172.21.15.22:0/1032117147 conn(0x7fb6a00617a0 msgr2=0x7fb6a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.386+0000 7fb6a6872640 1 -- 172.21.15.22:0/1032117147 shutdown_connections 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.386+0000 7fb6a6872640 1 -- 172.21.15.22:0/1032117147 wait complete. 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.387+0000 7fb6a6872640 10 monclient: build_initial_monmap 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.387+0000 7fb6a6872640 10 monclient: monmap: 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.388+0000 7fb6a6872640 1 librados: starting msgr at 2024-03-25T18:19:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.388+0000 7fb6a6872640 1 librados: starting objecter 2024-03-25T18:19:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.388+0000 7fb6a6872640 1 Processor -- start 2024-03-25T18:19:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.388+0000 7fb6a6872640 1 -- start start 2024-03-25T18:19:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.388+0000 7fb6a6872640 1 librados: setting wanted keys 2024-03-25T18:19:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.388+0000 7fb6a6872640 1 librados: calling monclient init 2024-03-25T18:19:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.388+0000 7fb6a6872640 10 monclient: init 2024-03-25T18:19:45.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.389+0000 7fb6a6872640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:45.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.389+0000 7fb6a6872640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:45.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.389+0000 7fb6a6872640 10 monclient(hunting): picked mon.noname-c con 0x7fb6a00b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:45.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb6a6872640 10 monclient(hunting): picked mon.noname-a con 0x7fb6a00b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:45.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb6a6872640 10 monclient(hunting): picked mon.noname-b con 0x7fb6a014be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:45.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb6a6872640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a013a7d0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb6a6872640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a013aab0 con 0x7fb6a014be90 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb6a6872640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a01d54f0 con 0x7fb6a00b1730 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb6a6872640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb6a6872640 10 monclient(hunting): authenticate will time out at 7184.614033s 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb69f7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb6a00b4d60 0x7fb6a0131e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46472/0 (socket says 172.21.15.22:46472) 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.390+0000 7fb69f7fe640 1 -- 172.21.15.22:0/3964040609 learned_addr learned my addr 172.21.15.22:0/3964040609 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.391+0000 7fb69cff9640 10 client.?.objecter ms_handle_connect 0x7fb6a00b4d60 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.391+0000 7fb69cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.391+0000 7fb69cff9640 10 client.?.objecter ms_handle_connect 0x7fb6a00b1730 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.391+0000 7fb69cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.391+0000 7fb69cff9640 10 client.?.objecter ms_handle_connect 0x7fb6a014be90 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.391+0000 7fb69cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.392+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484892828 0 0) 0x7fb6a013a7d0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.392+0000 7fb69cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.392+0000 7fb69cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.392+0000 7fb69cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.392+0000 7fb69cff9640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.392+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68c004eb0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397290656 0 0) 0x7fb6a013aab0 con 0x7fb6a014be90 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a013a7d0 con 0x7fb6a014be90 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770917293 0 0) 0x7fb6a01d54f0 con 0x7fb6a00b1730 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:19:45.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.394+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a013aab0 con 0x7fb6a00b1730 2024-03-25T18:19:45.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.399+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1585568627 0 0) 0x7fb68c004eb0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.399+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a01d54f0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.399+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb694001670 con 0x7fb6a00b4d60 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.399+0000 7fb69cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.399+0000 7fb69cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.399+0000 7fb69cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.399+0000 7fb69cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:45.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 850925377 0 0) 0x7fb6a013a7d0 con 0x7fb6a014be90 2024-03-25T18:19:45.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb68c004eb0 con 0x7fb6a014be90 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb698001a50 con 0x7fb6a014be90 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:45.403 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:45.404 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1692619113 0 0) 0x7fb6a013aab0 con 0x7fb6a00b1730 2024-03-25T18:19:45.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.401+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a013a7d0 con 0x7fb6a00b1730 2024-03-25T18:19:45.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.402+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb690001870 con 0x7fb6a00b1730 2024-03-25T18:19:45.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.402+0000 7fb69cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.402+0000 7fb69cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:45.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.402+0000 7fb69cff9640 10 monclient(hunting): dump: 2024-03-25T18:19:45.406 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:45.406 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2279863720 0 0) 0x7fb6a01d54f0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 >> v1:172.21.15.22:6790/0 conn(0x7fb6a00b1730 legacy=0x7fb6a013aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 >> v1:172.21.15.33:6789/0 conn(0x7fb6a014be90 legacy=0x7fb6a01e32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 1 monclient: found mon.a 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0139600 con 0x7fb6a00b4d60 2024-03-25T18:19:45.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb69cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 5 monclient: authenticate success, global_id 4218 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 10 monclient: _renew_subs 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a0139850 con 0x7fb6a00b4d60 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 10 monclient: _renew_subs 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a013b7c0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.403+0000 7fb6a6872640 1 librados: init done 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.404+0000 7fb6a6872640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.404+0000 7fb6a6872640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.404+0000 7fb6a6872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:45.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.404+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a0138220 con 0x7fb6a00b4d60 2024-03-25T18:19:45.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.406+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6940011d0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.406+0000 7fb69cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:45.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.406+0000 7fb69e7fc640 4 set_mon_vals no callback set 2024-03-25T18:19:45.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.407+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb694004b40 con 0x7fb6a00b4d60 2024-03-25T18:19:45.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.407+0000 7fb69cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.407+0000 7fb69cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.407+0000 7fb69cff9640 10 monclient: dump: 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:45.410 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:45.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.422+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fb69401ab90 con 0x7fb6a00b4d60 2024-03-25T18:19:45.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.423+0000 7fb69ffff640 10 monclient: get_auth_request con 0x7fb68c06e210 auth_method 0 2024-03-25T18:19:45.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.423+0000 7fb69cff9640 10 client.4218.objecter ms_handle_connect 0x7fb68c06e210 2024-03-25T18:19:45.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fb694054960 con 0x7fb6a00b4d60 2024-03-25T18:19:45.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 10 client.4218.objecter ms_dispatch 0x7fb6a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:45.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:45.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:45.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T18:19:45.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6940502a0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:45.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.672+0000 7fb69cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:45.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.937+0000 7fb6a6872640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.937+0000 7fb6a6872640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.937+0000 7fb6a6872640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:19:45.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.937+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6a00627b0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.979+0000 7fb69cff9640 1 -- 172.21.15.22:0/3964040609 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fb6940523a0 con 0x7fb6a00b4d60 2024-03-25T18:19:45.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.980+0000 7fb69cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:45.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.980+0000 7fb69cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:45.981 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:19:45.982 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:45.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.982+0000 7fb6a6872640 10 librados: watch_flush enter 2024-03-25T18:19:45.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 10 librados: watch_flush exit 2024-03-25T18:19:45.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T18:19:45.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 10 client.4218.objecter successfully canceled tick 2024-03-25T18:19:45.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 >> v1:172.21.15.33:6800/215350958 conn(0x7fb68c06e210 legacy=0x7fb68c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 10 monclient: shutdown 2024-03-25T18:19:45.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:45.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 >> v1:172.21.15.22:6789/0 conn(0x7fb6a00b4d60 legacy=0x7fb6a0131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:45.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 shutdown_connections 2024-03-25T18:19:45.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.983+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 >> 172.21.15.22:0/3964040609 conn(0x7fb6a00617a0 msgr2=0x7fb6a00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:45.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.984+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 shutdown_connections 2024-03-25T18:19:45.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.984+0000 7fb6a6872640 1 -- 172.21.15.22:0/3964040609 wait complete. 2024-03-25T18:19:45.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:45.984+0000 7fb6a6872640 1 librados: shutdown 2024-03-25T18:19:45.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:46.826 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:00:26.467 35193== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:47.075 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:00:26.717 35192== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:47.120 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:00:26.763 35191== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:47.335 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:00:26.971 35194== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:47.383 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:00:27.017 35450== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:47.677 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:00:27.305 35451== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:47.723 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:00:27.367 35449== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:47.983 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:00:27.610 35452== Warning: unimplemented fcntl command: 1038 2024-03-25T18:19:52.804 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.923+0000 7f74eb2c8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.923+0000 7f74eb2c8640 10 monclient: build_initial_monmap 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.923+0000 7f74eb2c8640 10 monclient: monmap: 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:19:52.924366+0000 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:19:52.924366+0000 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:52.924 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:19:52.925 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:19:52.925 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.923+0000 7f74eb2c8640 1 Processor -- start 2024-03-25T18:19:52.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.924+0000 7f74eb2c8640 1 -- start start 2024-03-25T18:19:52.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.924+0000 7f74eb2c8640 10 monclient: init 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f74e40a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f74e413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f74e41430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e4142520 con 0x7f74e41430c0 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e40b28a0 con 0x7f74e413f710 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e40b2a30 con 0x7f74e40a98f0 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.925+0000 7f74eb2c8640 10 monclient(hunting): authenticate will time out at 7192.149148s 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.926+0000 7f74e9064640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f74e40a98f0 0x7f74e40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:38062/0 (socket says 172.21.15.22:38062) 2024-03-25T18:19:52.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.926+0000 7f74e9064640 1 -- 172.21.15.22:0/1910091782 learned_addr learned my addr 172.21.15.22:0/1910091782 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:52.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838134273 0 0) 0x7f74e40b28a0 con 0x7f74e413f710 2024-03-25T18:19:52.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:52.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:52.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:52.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:19:52.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003520 con 0x7f74e413f710 2024-03-25T18:19:52.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789097296 0 0) 0x7f74e40b2a30 con 0x7f74e40a98f0 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e40b28a0 con 0x7f74e40a98f0 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290965160 0 0) 0x7f74e4142520 con 0x7f74e41430c0 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:19:52.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.930+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e40b2a30 con 0x7f74e41430c0 2024-03-25T18:19:52.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 634060294 0 0) 0x7f74d0003520 con 0x7f74e413f710 2024-03-25T18:19:52.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 >> v1:172.21.15.22:6790/0 conn(0x7f74e40a98f0 legacy=0x7f74e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:52.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 >> v1:172.21.15.22:6789/0 conn(0x7f74e41430c0 legacy=0x7f74e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:52.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 1 monclient: found mon.noname-c 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.934+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e40b44c0 con 0x7f74e413f710 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74dbfff640 10 monclient: _finish_auth 0 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74dbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74eb2c8640 5 monclient: authenticate success, global_id 4219 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74eb2c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f74d4001650 con 0x7f74e413f710 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74dbfff640 10 monclient: dump: 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:52.937 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.935+0000 7f74eb2c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:52.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.937+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74d40018a0 con 0x7f74e413f710 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.937+0000 7f74dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.937+0000 7f74eb2c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.937+0000 7f74eb2c8640 4 set_mon_vals no callback set 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74dbfff640 1 -- 172.21.15.22:0/1910091782 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f74d4001c30 con 0x7f74e413f710 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74dbfff640 10 monclient: dump: 2024-03-25T18:19:52.939 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74eb2c8640 10 monclient: shutdown 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74eb2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:52.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74eb2c8640 1 -- 172.21.15.22:0/1910091782 >> v1:172.21.15.33:6789/0 conn(0x7f74e413f710 legacy=0x7f74e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:52.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74eb2c8640 1 -- 172.21.15.22:0/1910091782 shutdown_connections 2024-03-25T18:19:52.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74eb2c8640 1 -- 172.21.15.22:0/1910091782 >> 172.21.15.22:0/1910091782 conn(0x7f74e40b7ea0 msgr2=0x7f74e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:52.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74eb2c8640 1 -- 172.21.15.22:0/1910091782 shutdown_connections 2024-03-25T18:19:52.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.938+0000 7f74eb2c8640 1 -- 172.21.15.22:0/1910091782 wait complete. 2024-03-25T18:19:52.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.939+0000 7f74eb2c8640 10 monclient: build_initial_monmap 2024-03-25T18:19:52.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.939+0000 7f74eb2c8640 10 monclient: monmap: 2024-03-25T18:19:52.941 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.939+0000 7f74eb2c8640 1 librados: starting msgr at 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.939+0000 7f74eb2c8640 1 librados: starting objecter 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.939+0000 7f74eb2c8640 1 Processor -- start 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.940+0000 7f74eb2c8640 1 -- start start 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.940+0000 7f74eb2c8640 1 librados: setting wanted keys 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.940+0000 7f74eb2c8640 1 librados: calling monclient init 2024-03-25T18:19:52.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.940+0000 7f74eb2c8640 10 monclient: init 2024-03-25T18:19:52.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.940+0000 7f74eb2c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:52.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.940+0000 7f74eb2c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f74e40a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f74e413f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f74e41430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e414c570 con 0x7f74e40a98f0 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e414da20 con 0x7f74e41430c0 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e414c000 con 0x7f74e413f710 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74eb2c8640 10 monclient(hunting): authenticate will time out at 7192.165260s 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74e9064640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f74e40a98f0 0x7f74e41dbba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46496/0 (socket says 172.21.15.22:46496) 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.941+0000 7f74e9064640 1 -- 172.21.15.22:0/56082270 learned_addr learned my addr 172.21.15.22:0/56082270 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.942+0000 7f74d9ffb640 10 client.?.objecter ms_handle_connect 0x7f74e40a98f0 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.942+0000 7f74d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.942+0000 7f74d9ffb640 10 client.?.objecter ms_handle_connect 0x7f74e413f710 2024-03-25T18:19:52.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.942+0000 7f74d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:52.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.942+0000 7f74d9ffb640 10 client.?.objecter ms_handle_connect 0x7f74e41430c0 2024-03-25T18:19:52.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.942+0000 7f74d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:52.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.944+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787333914 0 0) 0x7f74e414c000 con 0x7f74e413f710 2024-03-25T18:19:52.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c8004eb0 con 0x7f74e413f710 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632657720 0 0) 0x7f74e414da20 con 0x7f74e41430c0 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e414c000 con 0x7f74e41430c0 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657154885 0 0) 0x7f74e414c570 con 0x7f74e40a98f0 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:19:52.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.945+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e414da20 con 0x7f74e40a98f0 2024-03-25T18:19:52.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311747688 0 0) 0x7f74c8004eb0 con 0x7f74e413f710 2024-03-25T18:19:52.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74e414c570 con 0x7f74e413f710 2024-03-25T18:19:52.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f74d4001bb0 con 0x7f74e413f710 2024-03-25T18:19:52.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 246046047 0 0) 0x7f74e414c000 con 0x7f74e41430c0 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c8004eb0 con 0x7f74e41430c0 2024-03-25T18:19:52.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f74dc001b50 con 0x7f74e41430c0 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.952+0000 7f74d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.953+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1139112762 0 0) 0x7f74e414da20 con 0x7f74e40a98f0 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.953+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74e414c000 con 0x7f74e40a98f0 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.953+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f74e0001930 con 0x7f74e40a98f0 2024-03-25T18:19:52.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.953+0000 7f74d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.953+0000 7f74d9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.953+0000 7f74d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:52.958 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3049849380 0 0) 0x7f74e414c570 con 0x7f74e413f710 2024-03-25T18:19:52.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 >> v1:172.21.15.33:6789/0 conn(0x7f74e41430c0 legacy=0x7f74e41e65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:52.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 >> v1:172.21.15.22:6789/0 conn(0x7f74e40a98f0 legacy=0x7f74e41dbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:52.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:52.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 1 monclient: found mon.c 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e414c190 con 0x7f74e413f710 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74eb2c8640 5 monclient: authenticate success, global_id 4226 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74eb2c8640 10 monclient: _renew_subs 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74eb2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.958+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e41414a0 con 0x7f74e413f710 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.959+0000 7f74eb2c8640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.959+0000 7f74eb2c8640 10 monclient: _renew_subs 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.959+0000 7f74eb2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.959+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e40b71f0 con 0x7f74e413f710 2024-03-25T18:19:52.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.959+0000 7f74eb2c8640 1 librados: init done 2024-03-25T18:19:52.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.960+0000 7f74eb2c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:52.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.960+0000 7f74eb2c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:52.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.960+0000 7f74eb2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:52.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.960+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e40627b0 con 0x7f74e413f710 2024-03-25T18:19:52.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.987+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74d40038a0 con 0x7f74e413f710 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.987+0000 7f74d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.987+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f74d4003bb0 con 0x7f74e413f710 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.987+0000 7f74d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.987+0000 7f74d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.987+0000 7f74d9ffb640 10 monclient: dump: 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:19:52.989 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:19:52.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.987+0000 7f74db7fe640 4 set_mon_vals no callback set 2024-03-25T18:19:52.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.988+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f74d4019b80 con 0x7f74e413f710 2024-03-25T18:19:52.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.989+0000 7f74e9064640 10 monclient: get_auth_request con 0x7f74c806e040 auth_method 0 2024-03-25T18:19:52.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.990+0000 7f74d9ffb640 10 client.4226.objecter ms_handle_connect 0x7f74c806e040 2024-03-25T18:19:52.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.992+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f74d40536a0 con 0x7f74e413f710 2024-03-25T18:19:52.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.992+0000 7f74d9ffb640 10 client.4226.objecter ms_dispatch 0x7f74e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:52.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.992+0000 7f74d9ffb640 3 client.4226.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:52.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.992+0000 7f74d9ffb640 3 client.4226.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:52.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:52.992+0000 7f74d9ffb640 20 client.4226.objecter dump_active .. 0 homeless 2024-03-25T18:19:53.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.323+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74d404f1b0 con 0x7f74e413f710 2024-03-25T18:19:53.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.323+0000 7f74d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:53.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.323+0000 7f74d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:53.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.643+0000 7f74eb2c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:53.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.644+0000 7f74eb2c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:53.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.644+0000 7f74eb2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:19:53.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.644+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74e413e230 con 0x7f74e413f710 2024-03-25T18:19:53.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.671+0000 7f74d9ffb640 1 -- 172.21.15.22:0/56082270 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f74d40510e0 con 0x7f74e413f710 2024-03-25T18:19:53.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.671+0000 7f74d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:53.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.671+0000 7f74d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:53.672 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:19:53.672 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:53.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 10 librados: watch_flush enter 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 10 librados: watch_flush exit 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 20 client.4226.objecter shutdown clearing up homeless session... 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 10 client.4226.objecter successfully canceled tick 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 >> v1:172.21.15.33:6800/215350958 conn(0x7f74c806e040 legacy=0x7f74c8070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 10 monclient: shutdown 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 >> v1:172.21.15.22:6790/0 conn(0x7f74e413f710 legacy=0x7f74e41dc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 shutdown_connections 2024-03-25T18:19:53.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.674+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 >> 172.21.15.22:0/56082270 conn(0x7f74e40b7ea0 msgr2=0x7f74e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:53.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.675+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 shutdown_connections 2024-03-25T18:19:53.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.675+0000 7f74eb2c8640 1 -- 172.21.15.22:0/56082270 wait complete. 2024-03-25T18:19:53.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:19:53.675+0000 7f74eb2c8640 1 librados: shutdown 2024-03-25T18:19:53.691 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:55.132 INFO:tasks.ceph.osd.5.smithi033.stderr:2024-03-25T18:19:55.128+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:19:55.357 INFO:tasks.ceph.osd.4.smithi033.stderr:2024-03-25T18:19:55.354+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:19:55.547 INFO:tasks.ceph.osd.6.smithi033.stderr:2024-03-25T18:19:55.545+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:19:55.848 INFO:tasks.ceph.osd.7.smithi033.stderr:2024-03-25T18:19:55.846+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:19:56.179 INFO:tasks.ceph.osd.0.smithi022.stderr:2024-03-25T18:19:56.175+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:19:56.525 INFO:tasks.ceph.osd.2.smithi022.stderr:2024-03-25T18:19:56.520+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:19:56.699 INFO:tasks.ceph.osd.3.smithi022.stderr:2024-03-25T18:19:56.696+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:19:56.955 INFO:tasks.ceph.osd.1.smithi022.stderr:2024-03-25T18:19:56.950+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:20:00.497 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.609+0000 7efcfe967640 10 monclient: get_monmap_and_config 2024-03-25T18:20:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.609+0000 7efcfe967640 10 monclient: build_initial_monmap 2024-03-25T18:20:00.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.609+0000 7efcfe967640 10 monclient: monmap: 2024-03-25T18:20:00.610 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:00.610 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:00.610 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:00.610686+0000 2024-03-25T18:20:00.611 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:00.610686+0000 2024-03-25T18:20:00.611 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:00.611 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:00.611 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:00.611 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:00.611 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:00.611 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 1 Processor -- start 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 1 -- start start 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 10 monclient: init 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 10 monclient(hunting): picked mon.noname-c con 0x7efcf806c380 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 10 monclient(hunting): picked mon.noname-a con 0x7efcf80b12d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.610+0000 7efcfe967640 10 monclient(hunting): picked mon.noname-b con 0x7efcf80aedf0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.611+0000 7efcfe967640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80b16f0 con 0x7efcf80b12d0 2024-03-25T18:20:00.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.611+0000 7efcfe967640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80adfe0 con 0x7efcf806c380 2024-03-25T18:20:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.611+0000 7efcfe967640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf80add00 con 0x7efcf80aedf0 2024-03-25T18:20:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.611+0000 7efcfe967640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:00.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.611+0000 7efcfe967640 10 monclient(hunting): authenticate will time out at 7199.834624s 2024-03-25T18:20:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.627+0000 7efcf77fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7efcf80b12d0 0x7efcf80ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:51412/0 (socket says 172.21.15.22:51412) 2024-03-25T18:20:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.627+0000 7efcf77fe640 1 -- 172.21.15.22:0/949867084 learned_addr learned my addr 172.21.15.22:0/949867084 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.640+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643483213 0 0) 0x7efcf80b16f0 con 0x7efcf80b12d0 2024-03-25T18:20:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.640+0000 7efcf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.640+0000 7efcf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.640+0000 7efcf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.640+0000 7efcf6ffd640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:20:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.640+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce0003520 con 0x7efcf80b12d0 2024-03-25T18:20:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.645+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705143559 0 0) 0x7efcf80adfe0 con 0x7efcf806c380 2024-03-25T18:20:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.645+0000 7efcf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.645+0000 7efcf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.645+0000 7efcf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.645+0000 7efcf6ffd640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:20:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.645+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf80b16f0 con 0x7efcf806c380 2024-03-25T18:20:00.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.648+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477107581 0 0) 0x7efcf80add00 con 0x7efcf80aedf0 2024-03-25T18:20:00.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.648+0000 7efcf6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.648+0000 7efcf6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.648+0000 7efcf6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.648+0000 7efcf6ffd640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:20:00.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.648+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf80adfe0 con 0x7efcf80aedf0 2024-03-25T18:20:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243233136 0 0) 0x7efce0003520 con 0x7efcf80b12d0 2024-03-25T18:20:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 >> v1:172.21.15.22:6790/0 conn(0x7efcf80aedf0 legacy=0x7efcf814e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 >> v1:172.21.15.33:6789/0 conn(0x7efcf806c380 legacy=0x7efcf80b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:20:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf80ae2c0 con 0x7efcf80b12d0 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efcec001690 con 0x7efcf80b12d0 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcf6ffd640 10 monclient: dump: 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:00.657 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:00.658 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:00.658 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:00.658 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:00.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcfe967640 5 monclient: authenticate success, global_id 4251 2024-03-25T18:20:00.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.655+0000 7efcfe967640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.660+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcec0017f0 con 0x7efcf80b12d0 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.660+0000 7efcf6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.660+0000 7efcf6ffd640 1 -- 172.21.15.22:0/949867084 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efcec001b80 con 0x7efcf80b12d0 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.660+0000 7efcf6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.660+0000 7efcf6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.660+0000 7efcf6ffd640 10 monclient: dump: 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.662 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.663 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:00.663 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:00.663 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:00.663 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:00.663 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:00.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.661+0000 7efcfe967640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:00.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.661+0000 7efcfe967640 4 set_mon_vals no callback set 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.661+0000 7efcfe967640 10 monclient: shutdown 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.661+0000 7efcfe967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.661+0000 7efcfe967640 1 -- 172.21.15.22:0/949867084 >> v1:172.21.15.22:6789/0 conn(0x7efcf80b12d0 legacy=0x7efcf80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.662+0000 7efcfe967640 1 -- 172.21.15.22:0/949867084 shutdown_connections 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.662+0000 7efcfe967640 1 -- 172.21.15.22:0/949867084 >> 172.21.15.22:0/949867084 conn(0x7efcf80b5d10 msgr2=0x7efcf80b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.662+0000 7efcfe967640 1 -- 172.21.15.22:0/949867084 shutdown_connections 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.662+0000 7efcfe967640 1 -- 172.21.15.22:0/949867084 wait complete. 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 10 monclient: build_initial_monmap 2024-03-25T18:20:00.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 10 monclient: monmap: 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 1 librados: starting msgr at 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 1 librados: starting objecter 2024-03-25T18:20:00.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 1 Processor -- start 2024-03-25T18:20:00.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 1 -- start start 2024-03-25T18:20:00.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 1 librados: setting wanted keys 2024-03-25T18:20:00.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 1 librados: calling monclient init 2024-03-25T18:20:00.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.663+0000 7efcfe967640 10 monclient: init 2024-03-25T18:20:00.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:00.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:00.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 10 monclient(hunting): picked mon.noname-c con 0x7efcf806c380 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:00.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 10 monclient(hunting): picked mon.noname-a con 0x7efcf80aedf0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:00.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 10 monclient(hunting): picked mon.noname-b con 0x7efcf80b12d0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:00.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf806c8e0 con 0x7efcf80aedf0 2024-03-25T18:20:00.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf81db650 con 0x7efcf80b12d0 2024-03-25T18:20:00.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf81c7460 con 0x7efcf806c380 2024-03-25T18:20:00.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:00.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.664+0000 7efcfe967640 10 monclient(hunting): authenticate will time out at 7199.888027s 2024-03-25T18:20:00.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.682+0000 7efcf7fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7efcf806c380 0x7efcf81dab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60902/0 (socket says 172.21.15.22:60902) 2024-03-25T18:20:00.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.682+0000 7efcf7fff640 1 -- 172.21.15.22:0/2775990640 learned_addr learned my addr 172.21.15.22:0/2775990640 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:00.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.686+0000 7efcf4ff9640 10 client.?.objecter ms_handle_connect 0x7efcf806c380 2024-03-25T18:20:00.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.686+0000 7efcf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:00.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.687+0000 7efcf4ff9640 10 client.?.objecter ms_handle_connect 0x7efcf80b12d0 2024-03-25T18:20:00.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.687+0000 7efcf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:00.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.693+0000 7efcf4ff9640 10 client.?.objecter ms_handle_connect 0x7efcf80aedf0 2024-03-25T18:20:00.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.693+0000 7efcf4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:00.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.701+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229123463 0 0) 0x7efcf806c8e0 con 0x7efcf80aedf0 2024-03-25T18:20:00.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.701+0000 7efcf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.701+0000 7efcf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.701+0000 7efcf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.701+0000 7efcf4ff9640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:20:00.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.701+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efce4004eb0 con 0x7efcf80aedf0 2024-03-25T18:20:00.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.702+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536107387 0 0) 0x7efcf81db650 con 0x7efcf80b12d0 2024-03-25T18:20:00.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.702+0000 7efcf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.702+0000 7efcf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.702+0000 7efcf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.702+0000 7efcf4ff9640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:20:00.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.702+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf806c8e0 con 0x7efcf80b12d0 2024-03-25T18:20:00.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.704+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090351467 0 0) 0x7efcf81c7460 con 0x7efcf806c380 2024-03-25T18:20:00.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.704+0000 7efcf4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:00.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.704+0000 7efcf4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:00.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.704+0000 7efcf4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:00.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.704+0000 7efcf4ff9640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:20:00.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.704+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf81db650 con 0x7efcf806c380 2024-03-25T18:20:00.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.717+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2046598616 0 0) 0x7efce4004eb0 con 0x7efcf80aedf0 2024-03-25T18:20:00.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.717+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcf81c7460 con 0x7efcf80aedf0 2024-03-25T18:20:00.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.717+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efcec001b00 con 0x7efcf80aedf0 2024-03-25T18:20:00.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.717+0000 7efcf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.717+0000 7efcf4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.717+0000 7efcf4ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:00.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.717+0000 7efcf4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:00.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.719+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 280827436 0 0) 0x7efcf806c8e0 con 0x7efcf80b12d0 2024-03-25T18:20:00.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.719+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efce4004eb0 con 0x7efcf80b12d0 2024-03-25T18:20:00.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.719+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efcf0001bf0 con 0x7efcf80b12d0 2024-03-25T18:20:00.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.719+0000 7efcf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.719+0000 7efcf4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:00.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.719+0000 7efcf4ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:00.721 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.730+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2839344836 0 0) 0x7efcf81db650 con 0x7efcf806c380 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.730+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcf806c8e0 con 0x7efcf806c380 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.730+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efce8001930 con 0x7efcf806c380 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.730+0000 7efcf4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.730+0000 7efcf4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.730+0000 7efcf4ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:00.731 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:00.732 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1399443268 0 0) 0x7efce4004eb0 con 0x7efcf80b12d0 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 >> v1:172.21.15.22:6790/0 conn(0x7efcf806c380 legacy=0x7efcf81dab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 >> v1:172.21.15.22:6789/0 conn(0x7efcf80aedf0 legacy=0x7efcf81a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 1 monclient: found mon.b 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf81db290 con 0x7efcf80b12d0 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcf4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:20:00.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 10 monclient: _renew_subs 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf81c4fd0 con 0x7efcf80b12d0 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 10 monclient: _renew_subs 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf81e30f0 con 0x7efcf80b12d0 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.735+0000 7efcfe967640 1 librados: init done 2024-03-25T18:20:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.736+0000 7efcfe967640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:00.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.736+0000 7efcfe967640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:00.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.736+0000 7efcfe967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:00.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:00.736+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf80627b0 con 0x7efcf80b12d0 2024-03-25T18:20:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.017+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efcf0002080 con 0x7efcf80b12d0 2024-03-25T18:20:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.017+0000 7efcf4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.017+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efcf0003b30 con 0x7efcf80b12d0 2024-03-25T18:20:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.017+0000 7efcf4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.017+0000 7efcf4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.017+0000 7efcf4ff9640 10 monclient: dump: 2024-03-25T18:20:01.018 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:01.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.017+0000 7efcf67fc640 4 set_mon_vals no callback set 2024-03-25T18:20:01.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.018+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7efcf0019b00 con 0x7efcf80b12d0 2024-03-25T18:20:01.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.018+0000 7efcf7fff640 10 monclient: get_auth_request con 0x7efce406e130 auth_method 0 2024-03-25T18:20:01.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.018+0000 7efcf4ff9640 10 client.4252.objecter ms_handle_connect 0x7efce406e130 2024-03-25T18:20:01.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.024+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7efcf00537f0 con 0x7efcf80b12d0 2024-03-25T18:20:01.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.024+0000 7efcf4ff9640 10 client.4252.objecter ms_dispatch 0x7efcf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:01.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.024+0000 7efcf4ff9640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:01.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.024+0000 7efcf4ff9640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:01.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.024+0000 7efcf4ff9640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T18:20:01.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.025+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efcf004f130 con 0x7efcf80b12d0 2024-03-25T18:20:01.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.025+0000 7efcf4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:01.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.025+0000 7efcf4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:01.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.313+0000 7efcfe967640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.313+0000 7efcfe967640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.313+0000 7efcfe967640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.313+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcf81db420 con 0x7efcf80b12d0 2024-03-25T18:20:01.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.360+0000 7efcf4ff9640 1 -- 172.21.15.22:0/2775990640 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7efcf81db420 con 0x7efcf80b12d0 2024-03-25T18:20:01.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.360+0000 7efcf4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:01.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.360+0000 7efcf4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:01.361 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:01.361 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 10 librados: watch_flush enter 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 10 librados: watch_flush exit 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 10 client.4252.objecter successfully canceled tick 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 >> v1:172.21.15.33:6800/215350958 conn(0x7efce406e130 legacy=0x7efce40705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 10 monclient: shutdown 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 >> v1:172.21.15.33:6789/0 conn(0x7efcf80b12d0 legacy=0x7efcf81a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 shutdown_connections 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 >> 172.21.15.22:0/2775990640 conn(0x7efcf80b5d10 msgr2=0x7efcf81cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:01.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 shutdown_connections 2024-03-25T18:20:01.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 1 -- 172.21.15.22:0/2775990640 wait complete. 2024-03-25T18:20:01.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:01.362+0000 7efcfe967640 1 librados: shutdown 2024-03-25T18:20:01.374 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:08.180 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.298+0000 7f72e8a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.298+0000 7f72e8a4a640 10 monclient: build_initial_monmap 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.298+0000 7f72e8a4a640 10 monclient: monmap: 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:08.299268+0000 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:08.299268+0000 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:08.299 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.299+0000 7f72e8a4a640 1 Processor -- start 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.299+0000 7f72e8a4a640 1 -- start start 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.299+0000 7f72e8a4a640 10 monclient: init 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f72e406e200 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f72e4072730 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f72e406e5e0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40ae470 con 0x7f72e406e5e0 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40b0320 con 0x7f72e406e200 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e406efa0 con 0x7f72e4072730 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:08.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.300+0000 7f72e8a4a640 10 monclient(hunting): authenticate will time out at 7207.523412s 2024-03-25T18:20:08.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.308+0000 7f72e259c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f72e406e200 0x7f72e4072020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:32836/0 (socket says 172.21.15.22:32836) 2024-03-25T18:20:08.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.308+0000 7f72e259c640 1 -- 172.21.15.22:0/3082199790 learned_addr learned my addr 172.21.15.22:0/3082199790 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:08.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.330+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016871443 0 0) 0x7f72e40b0320 con 0x7f72e406e200 2024-03-25T18:20:08.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.330+0000 7f72e159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.330+0000 7f72e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.330+0000 7f72e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.330+0000 7f72e159a640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:20:08.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.330+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72cc003520 con 0x7f72e406e200 2024-03-25T18:20:08.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.336+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193264620 0 0) 0x7f72e406efa0 con 0x7f72e4072730 2024-03-25T18:20:08.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.336+0000 7f72e159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.336+0000 7f72e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.336+0000 7f72e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.336+0000 7f72e159a640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:20:08.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.336+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e40b0320 con 0x7f72e4072730 2024-03-25T18:20:08.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.338+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808884478 0 0) 0x7f72e40ae470 con 0x7f72e406e5e0 2024-03-25T18:20:08.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.338+0000 7f72e159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.338+0000 7f72e159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.338+0000 7f72e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.338+0000 7f72e159a640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:20:08.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.338+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e406efa0 con 0x7f72e406e5e0 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1038455662 0 0) 0x7f72cc003520 con 0x7f72e406e200 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 >> v1:172.21.15.22:6790/0 conn(0x7f72e4072730 legacy=0x7f72e4146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 >> v1:172.21.15.22:6789/0 conn(0x7f72e406e5e0 legacy=0x7f72e4152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 1 monclient: found mon.noname-c 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e406f3f0 con 0x7f72e406e200 2024-03-25T18:20:08.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 10 monclient: _finish_auth 0 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f72d4001690 con 0x7f72e406e200 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e159a640 10 monclient: dump: 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:08.347 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:08.348 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:08.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e8a4a640 5 monclient: authenticate success, global_id 4255 2024-03-25T18:20:08.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.346+0000 7f72e8a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72d40017f0 con 0x7f72e406e200 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e159a640 1 -- 172.21.15.22:0/3082199790 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f72d4001c00 con 0x7f72e406e200 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e159a640 10 monclient: dump: 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:08.368 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e8a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:08.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.368+0000 7f72e8a4a640 4 set_mon_vals no callback set 2024-03-25T18:20:08.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.369+0000 7f72e8a4a640 10 monclient: shutdown 2024-03-25T18:20:08.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.369+0000 7f72e8a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:08.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.369+0000 7f72e8a4a640 1 -- 172.21.15.22:0/3082199790 >> v1:172.21.15.33:6789/0 conn(0x7f72e406e200 legacy=0x7f72e4072020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.369+0000 7f72e8a4a640 1 -- 172.21.15.22:0/3082199790 shutdown_connections 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.369+0000 7f72e8a4a640 1 -- 172.21.15.22:0/3082199790 >> 172.21.15.22:0/3082199790 conn(0x7f72e406b630 msgr2=0x7f72e40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.369+0000 7f72e8a4a640 1 -- 172.21.15.22:0/3082199790 shutdown_connections 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.369+0000 7f72e8a4a640 1 -- 172.21.15.22:0/3082199790 wait complete. 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.370+0000 7f72e8a4a640 10 monclient: build_initial_monmap 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.370+0000 7f72e8a4a640 10 monclient: monmap: 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:08.371 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:08.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 librados: starting msgr at 2024-03-25T18:20:08.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 librados: starting objecter 2024-03-25T18:20:08.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 Processor -- start 2024-03-25T18:20:08.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 -- start start 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 librados: setting wanted keys 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 librados: calling monclient init 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient: init 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f72e406e200 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f72e406e5e0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f72e4072730 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e413a4d0 con 0x7f72e406e200 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e413a220 con 0x7f72e4072730 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e414a340 con 0x7f72e406e5e0 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:08.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.371+0000 7f72e8a4a640 10 monclient(hunting): authenticate will time out at 7207.594460s 2024-03-25T18:20:08.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.381+0000 7f72e259c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f72e406e200 0x7f72e4139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36556/0 (socket says 172.21.15.22:36556) 2024-03-25T18:20:08.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.381+0000 7f72e259c640 1 -- 172.21.15.22:0/746403426 learned_addr learned my addr 172.21.15.22:0/746403426 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.389+0000 7f72d2ffd640 10 client.?.objecter ms_handle_connect 0x7f72e406e200 2024-03-25T18:20:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.389+0000 7f72d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:08.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.390+0000 7f72d2ffd640 10 client.?.objecter ms_handle_connect 0x7f72e406e5e0 2024-03-25T18:20:08.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.390+0000 7f72d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:08.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.391+0000 7f72d2ffd640 10 client.?.objecter ms_handle_connect 0x7f72e4072730 2024-03-25T18:20:08.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.391+0000 7f72d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:08.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.398+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066271725 0 0) 0x7f72e413a220 con 0x7f72e4072730 2024-03-25T18:20:08.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.398+0000 7f72d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.398+0000 7f72d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.398+0000 7f72d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.398+0000 7f72d2ffd640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:20:08.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.398+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c4004eb0 con 0x7f72e4072730 2024-03-25T18:20:08.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233826457 0 0) 0x7f72e413a4d0 con 0x7f72e406e200 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e413a220 con 0x7f72e406e200 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252852453 0 0) 0x7f72e414a340 con 0x7f72e406e5e0 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:20:08.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.399+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e413a4d0 con 0x7f72e406e5e0 2024-03-25T18:20:08.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.422+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665510364 0 0) 0x7f72e413a220 con 0x7f72e406e200 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.422+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72e414a340 con 0x7f72e406e200 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.422+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f72d4001b40 con 0x7f72e406e200 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.422+0000 7f72d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.422+0000 7f72d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.422+0000 7f72d2ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:08.423 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:08.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.422+0000 7f72d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.433+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3614887214 0 0) 0x7f72c4004eb0 con 0x7f72e4072730 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.433+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72e413a220 con 0x7f72e4072730 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.433+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f72dc001c70 con 0x7f72e4072730 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.433+0000 7f72d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.433+0000 7f72d2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.433+0000 7f72d2ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.433 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:08.434 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:08.434 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:08.434 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:08.434 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:08.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 484416510 0 0) 0x7f72e413a220 con 0x7f72e4072730 2024-03-25T18:20:08.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 >> v1:172.21.15.22:6790/0 conn(0x7f72e406e5e0 legacy=0x7f72e4139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 >> v1:172.21.15.22:6789/0 conn(0x7f72e406e200 legacy=0x7f72e4139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:08.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:08.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 1 monclient: found mon.b 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e413a960 con 0x7f72e4072730 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 10 monclient: _renew_subs 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e4138110 con 0x7f72e4072730 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 10 monclient: _renew_subs 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e414a940 con 0x7f72e4072730 2024-03-25T18:20:08.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.440+0000 7f72e8a4a640 1 librados: init done 2024-03-25T18:20:08.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.441+0000 7f72e8a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:08.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.441+0000 7f72e8a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:08.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.441+0000 7f72e8a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:08.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.441+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e414abf0 con 0x7f72e4072730 2024-03-25T18:20:08.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.742+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72dc003990 con 0x7f72e4072730 2024-03-25T18:20:08.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.742+0000 7f72d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:08.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.742+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f72dc002100 con 0x7f72e4072730 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.742+0000 7f72d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.742+0000 7f72d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.742+0000 7f72d2ffd640 10 monclient: dump: 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:08.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.742+0000 7f72e0d99640 4 set_mon_vals no callback set 2024-03-25T18:20:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.743+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f72dc019b90 con 0x7f72e4072730 2024-03-25T18:20:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.744+0000 7f72e259c640 10 monclient: get_auth_request con 0x7f72c406dfb0 auth_method 0 2024-03-25T18:20:08.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.744+0000 7f72d2ffd640 10 client.4258.objecter ms_handle_connect 0x7f72c406dfb0 2024-03-25T18:20:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.745+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f72dc053990 con 0x7f72e4072730 2024-03-25T18:20:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.745+0000 7f72d2ffd640 10 client.4258.objecter ms_dispatch 0x7f72e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.745+0000 7f72d2ffd640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.745+0000 7f72d2ffd640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:08.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.745+0000 7f72d2ffd640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T18:20:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.749+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72dc04f2d0 con 0x7f72e4072730 2024-03-25T18:20:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.749+0000 7f72d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:08.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.749+0000 7f72d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:08.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.994+0000 7f72e8a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:08.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.994+0000 7f72e8a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:08.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.994+0000 7f72e8a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:20:08.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:08.994+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72e414a140 con 0x7f72e4072730 2024-03-25T18:20:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.131+0000 7f72d2ffd640 1 -- 172.21.15.22:0/746403426 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f72dc0513d0 con 0x7f72e4072730 2024-03-25T18:20:09.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.131+0000 7f72d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.131+0000 7f72d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.133+0000 7f72e8a4a640 10 librados: watch_flush enter 2024-03-25T18:20:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 10 librados: watch_flush exit 2024-03-25T18:20:09.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 10 client.4258.objecter successfully canceled tick 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 >> v1:172.21.15.33:6800/215350958 conn(0x7f72c406dfb0 legacy=0x7f72c4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 10 monclient: shutdown 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 >> v1:172.21.15.33:6789/0 conn(0x7f72e4072730 legacy=0x7f72e41e66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 shutdown_connections 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 >> 172.21.15.22:0/746403426 conn(0x7f72e406b630 msgr2=0x7f72e40aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 shutdown_connections 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 1 -- 172.21.15.22:0/746403426 wait complete. 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:09.134+0000 7f72e8a4a640 1 librados: shutdown 2024-03-25T18:20:09.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:12.000 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:14.662 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.064 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:14.726 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.174 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:14.836 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.236 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:14.898 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.339 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:14.957 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.381 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:15.043 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.411 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:15.030 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.517 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:15.179 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:12.636 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:15.253 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.952 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:16.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.069+0000 7f9f53fa4640 10 monclient: get_monmap_and_config 2024-03-25T18:20:16.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.069+0000 7f9f53fa4640 10 monclient: build_initial_monmap 2024-03-25T18:20:16.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.069+0000 7f9f53fa4640 10 monclient: monmap: 2024-03-25T18:20:16.069 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:16.069 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:16.069 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:16.069356+0000 2024-03-25T18:20:16.069 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:16.069356+0000 2024-03-25T18:20:16.070 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:16.070 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:16.070 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:16.070 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:16.070 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:16.070 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:16.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.069+0000 7f9f53fa4640 1 Processor -- start 2024-03-25T18:20:16.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.069+0000 7f9f53fa4640 1 -- start start 2024-03-25T18:20:16.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.069+0000 7f9f53fa4640 10 monclient: init 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 10 monclient(hunting): picked mon.noname-c con 0x7f9f4c0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 10 monclient(hunting): picked mon.noname-a con 0x7f9f4c13f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 10 monclient(hunting): picked mon.noname-b con 0x7f9f4c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c142520 con 0x7f9f4c13f710 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c0b28a0 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c0b2a30 con 0x7f9f4c1430c0 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:16.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.070+0000 7f9f53fa4640 10 monclient(hunting): authenticate will time out at 7215.293569s 2024-03-25T18:20:16.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.084+0000 7f9f51d40640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f9f4c0a98f0 0x7f9f4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:40918/0 (socket says 172.21.15.22:40918) 2024-03-25T18:20:16.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.084+0000 7f9f51d40640 1 -- 172.21.15.22:0/721195751 learned_addr learned my addr 172.21.15.22:0/721195751 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.093+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501844314 0 0) 0x7f9f4c0b28a0 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.093+0000 7f9f50d3e640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.093+0000 7f9f50d3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.093+0000 7f9f50d3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.093+0000 7f9f50d3e640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:20:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.094+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f38003520 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.103+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778191538 0 0) 0x7f9f4c142520 con 0x7f9f4c13f710 2024-03-25T18:20:16.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.103+0000 7f9f50d3e640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.103+0000 7f9f50d3e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.103+0000 7f9f50d3e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.103+0000 7f9f50d3e640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:20:16.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.103+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c0b28a0 con 0x7f9f4c13f710 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 94564127 0 0) 0x7f9f38003520 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 >> v1:172.21.15.22:6790/0 conn(0x7f9f4c1430c0 legacy=0x7f9f4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 >> v1:172.21.15.22:6789/0 conn(0x7f9f4c13f710 legacy=0x7f9f4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 10 monclient: _finish_hunting 0 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 1 monclient: found mon.noname-c 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c0b44c0 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 10 monclient: _finish_auth 0 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 10 monclient: _check_auth_tickets 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:16.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f48001690 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f53fa4640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f53fa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f50d3e640 10 monclient: dump: 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:16.111 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:16.112 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:16.112 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:16.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.109+0000 7f9f53fa4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:16.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f480017f0 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f50d3e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f50d3e640 1 -- 172.21.15.22:0/721195751 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f48001c00 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f53fa4640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f53fa4640 4 set_mon_vals no callback set 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f50d3e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f50d3e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.128+0000 7f9f50d3e640 10 monclient: dump: 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.129 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.130 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.130 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:16.130 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:16.130 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:16.130 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:16.130 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:16.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.129+0000 7f9f53fa4640 10 monclient: shutdown 2024-03-25T18:20:16.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.129+0000 7f9f53fa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:16.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.129+0000 7f9f53fa4640 1 -- 172.21.15.22:0/721195751 >> v1:172.21.15.33:6789/0 conn(0x7f9f4c0a98f0 legacy=0x7f9f4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.129+0000 7f9f53fa4640 1 -- 172.21.15.22:0/721195751 shutdown_connections 2024-03-25T18:20:16.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.129+0000 7f9f53fa4640 1 -- 172.21.15.22:0/721195751 >> 172.21.15.22:0/721195751 conn(0x7f9f4c0b7ea0 msgr2=0x7f9f4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:16.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.129+0000 7f9f53fa4640 1 -- 172.21.15.22:0/721195751 shutdown_connections 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.129+0000 7f9f53fa4640 1 -- 172.21.15.22:0/721195751 wait complete. 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.130+0000 7f9f53fa4640 10 monclient: build_initial_monmap 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.130+0000 7f9f53fa4640 10 monclient: monmap: 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.130+0000 7f9f53fa4640 1 librados: starting msgr at 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.130+0000 7f9f53fa4640 1 librados: starting objecter 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 1 Processor -- start 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 1 -- start start 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 1 librados: setting wanted keys 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 1 librados: calling monclient init 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient: init 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient(hunting): picked mon.noname-a con 0x7f9f4c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient(hunting): picked mon.noname-b con 0x7f9f4c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient(hunting): picked mon.noname-c con 0x7f9f4c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c138b40 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c138e20 con 0x7f9f4c13f710 2024-03-25T18:20:16.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4c139100 con 0x7f9f4c1430c0 2024-03-25T18:20:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:16.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.131+0000 7f9f53fa4640 10 monclient(hunting): authenticate will time out at 7215.354485s 2024-03-25T18:20:16.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.147+0000 7f9f52541640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f9f4c1430c0 0x7f9f4c1e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59656/0 (socket says 172.21.15.22:59656) 2024-03-25T18:20:16.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.147+0000 7f9f52541640 1 -- 172.21.15.22:0/4245950493 learned_addr learned my addr 172.21.15.22:0/4245950493 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:16.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.147+0000 7f9f427fc640 10 client.?.objecter ms_handle_connect 0x7f9f4c1430c0 2024-03-25T18:20:16.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.147+0000 7f9f427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:16.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.151+0000 7f9f427fc640 10 client.?.objecter ms_handle_connect 0x7f9f4c0a98f0 2024-03-25T18:20:16.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.151+0000 7f9f427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:16.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.161+0000 7f9f427fc640 10 client.?.objecter ms_handle_connect 0x7f9f4c13f710 2024-03-25T18:20:16.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.161+0000 7f9f427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:16.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.162+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2743979493 0 0) 0x7f9f4c138b40 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.163+0000 7f9f427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.163+0000 7f9f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.163+0000 7f9f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.163+0000 7f9f427fc640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:20:16.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.163+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f30004eb0 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.166+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1977324925 0 0) 0x7f9f4c139100 con 0x7f9f4c1430c0 2024-03-25T18:20:16.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.166+0000 7f9f427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.166+0000 7f9f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.166+0000 7f9f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.166+0000 7f9f427fc640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:20:16.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.166+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c138b40 con 0x7f9f4c1430c0 2024-03-25T18:20:16.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.173+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3830566836 0 0) 0x7f9f4c138b40 con 0x7f9f4c1430c0 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.173+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f4c139100 con 0x7f9f4c1430c0 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.173+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f44001bd0 con 0x7f9f4c1430c0 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.173+0000 7f9f427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.173+0000 7f9f427fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.173+0000 7f9f427fc640 10 monclient(hunting): dump: 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:16.174 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:16.175 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:16.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.173+0000 7f9f427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:16.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.176+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822733123 0 0) 0x7f9f4c138e20 con 0x7f9f4c13f710 2024-03-25T18:20:16.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.176+0000 7f9f427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.176+0000 7f9f427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.176+0000 7f9f427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.176+0000 7f9f427fc640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:20:16.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.176+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f4c138b40 con 0x7f9f4c13f710 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.180+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1240191004 0 0) 0x7f9f30004eb0 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.180+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f4c138e20 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.180+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f48001bc0 con 0x7f9f4c0a98f0 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.180+0000 7f9f427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.180+0000 7f9f427fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.180+0000 7f9f427fc640 10 monclient(hunting): dump: 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:16.181 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:16.182 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:16.182 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:16.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3281656369 0 0) 0x7f9f4c139100 con 0x7f9f4c1430c0 2024-03-25T18:20:16.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 >> v1:172.21.15.33:6789/0 conn(0x7f9f4c13f710 legacy=0x7f9f4c137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 >> v1:172.21.15.22:6789/0 conn(0x7f9f4c0a98f0 legacy=0x7f9f4c137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:16.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 1 monclient: found mon.c 2024-03-25T18:20:16.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c1397a0 con 0x7f9f4c1430c0 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f53fa4640 5 monclient: authenticate success, global_id 4268 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f53fa4640 10 monclient: _renew_subs 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f53fa4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.186+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4c136aa0 con 0x7f9f4c1430c0 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.187+0000 7f9f53fa4640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.187+0000 7f9f53fa4640 10 monclient: _renew_subs 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.187+0000 7f9f53fa4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.187+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4c13c320 con 0x7f9f4c1430c0 2024-03-25T18:20:16.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.187+0000 7f9f53fa4640 1 librados: init done 2024-03-25T18:20:16.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.187+0000 7f9f53fa4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:16.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.188+0000 7f9f53fa4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:16.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.188+0000 7f9f53fa4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:16.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.188+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4c1395b0 con 0x7f9f4c1430c0 2024-03-25T18:20:16.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.524+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f440020a0 con 0x7f9f4c1430c0 2024-03-25T18:20:16.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.524+0000 7f9f427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:16.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.524+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f44003b30 con 0x7f9f4c1430c0 2024-03-25T18:20:16.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.524+0000 7f9f427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.524+0000 7f9f427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:16.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.524+0000 7f9f427fc640 10 monclient: dump: 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:16.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.524+0000 7f9f43fff640 4 set_mon_vals no callback set 2024-03-25T18:20:16.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.525+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f9f44003d50 con 0x7f9f4c1430c0 2024-03-25T18:20:16.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.526+0000 7f9f51d40640 10 monclient: get_auth_request con 0x7f9f3006df20 auth_method 0 2024-03-25T18:20:16.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.526+0000 7f9f427fc640 10 client.4268.objecter ms_handle_connect 0x7f9f3006df20 2024-03-25T18:20:16.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f9f44054a30 con 0x7f9f4c1430c0 2024-03-25T18:20:16.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 10 client.4268.objecter ms_dispatch 0x7f9f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:16.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:16.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:16.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-25T18:20:16.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f44050370 con 0x7f9f4c1430c0 2024-03-25T18:20:16.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:16.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.531+0000 7f9f427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:16.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.792+0000 7f9f53fa4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:16.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.792+0000 7f9f53fa4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:16.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.792+0000 7f9f53fa4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:16.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.792+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f4c0627b0 con 0x7f9f4c1430c0 2024-03-25T18:20:16.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.947+0000 7f9f427fc640 1 -- 172.21.15.22:0/4245950493 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f9f44052470 con 0x7f9f4c1430c0 2024-03-25T18:20:16.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.947+0000 7f9f427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:16.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.947+0000 7f9f427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:16.948 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:16.948 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 10 librados: watch_flush enter 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 10 librados: watch_flush exit 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 10 client.4268.objecter successfully canceled tick 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 >> v1:172.21.15.33:6800/215350958 conn(0x7f9f3006df20 legacy=0x7f9f300703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 10 monclient: shutdown 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 >> v1:172.21.15.22:6790/0 conn(0x7f9f4c1430c0 legacy=0x7f9f4c1e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 shutdown_connections 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 >> 172.21.15.22:0/4245950493 conn(0x7f9f4c0b7ea0 msgr2=0x7f9f4c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 shutdown_connections 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 1 -- 172.21.15.22:0/4245950493 wait complete. 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:16.950+0000 7f9f53fa4640 1 librados: shutdown 2024-03-25T18:20:16.962 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:23.767 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.882+0000 7faa7b340640 10 monclient: get_monmap_and_config 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.882+0000 7faa7b340640 10 monclient: build_initial_monmap 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.882+0000 7faa7b340640 10 monclient: monmap: 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:23.882720+0000 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:23.882720+0000 2024-03-25T18:20:23.882 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.882+0000 7faa7b340640 1 Processor -- start 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.883+0000 7faa7b340640 1 -- start start 2024-03-25T18:20:23.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.883+0000 7faa7b340640 10 monclient: init 2024-03-25T18:20:23.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 10 monclient(hunting): picked mon.noname-b con 0x7faa740a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 10 monclient(hunting): picked mon.noname-a con 0x7faa7413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 10 monclient(hunting): picked mon.noname-c con 0x7faa741430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa74142520 con 0x7faa7413f710 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b28a0 con 0x7faa741430c0 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740b2a30 con 0x7faa740a98f0 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa7b340640 10 monclient(hunting): authenticate will time out at 7223.106924s 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa798dd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7faa741430c0 0x7faa7414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:48370/0 (socket says 172.21.15.22:48370) 2024-03-25T18:20:23.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.884+0000 7faa798dd640 1 -- 172.21.15.22:0/2090965100 learned_addr learned my addr 172.21.15.22:0/2090965100 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:23.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.960+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236816676 0 0) 0x7faa74142520 con 0x7faa7413f710 2024-03-25T18:20:23.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.960+0000 7faa6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:23.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.960+0000 7faa6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:23.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.960+0000 7faa6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:23.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.960+0000 7faa6bfff640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:20:23.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.960+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60003520 con 0x7faa7413f710 2024-03-25T18:20:23.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.962+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128335083 0 0) 0x7faa740b2a30 con 0x7faa740a98f0 2024-03-25T18:20:23.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.962+0000 7faa6bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:23.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.962+0000 7faa6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:23.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.962+0000 7faa6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:23.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.962+0000 7faa6bfff640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:20:23.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.962+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa74142520 con 0x7faa740a98f0 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 7278018 0 0) 0x7faa74142520 con 0x7faa740a98f0 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 >> v1:172.21.15.33:6789/0 conn(0x7faa741430c0 legacy=0x7faa7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 >> v1:172.21.15.22:6789/0 conn(0x7faa7413f710 legacy=0x7faa741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 1 monclient: found mon.noname-b 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa740b44c0 con 0x7faa740a98f0 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 10 monclient: _finish_auth 0 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa70001670 con 0x7faa740a98f0 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa7b340640 5 monclient: authenticate success, global_id 4271 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:23.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa6bfff640 10 monclient: dump: 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:23.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.970+0000 7faa7b340640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa700017d0 con 0x7faa740a98f0 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa6bfff640 1 -- 172.21.15.22:0/2090965100 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa70001ba0 con 0x7faa740a98f0 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa6bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa6bfff640 10 monclient: dump: 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:23.985 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa7b340640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:23.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.985+0000 7faa7b340640 4 set_mon_vals no callback set 2024-03-25T18:20:23.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.986+0000 7faa7b340640 10 monclient: shutdown 2024-03-25T18:20:23.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.986+0000 7faa7b340640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:23.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.986+0000 7faa7b340640 1 -- 172.21.15.22:0/2090965100 >> v1:172.21.15.22:6790/0 conn(0x7faa740a98f0 legacy=0x7faa740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:23.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.986+0000 7faa7b340640 1 -- 172.21.15.22:0/2090965100 shutdown_connections 2024-03-25T18:20:23.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.986+0000 7faa7b340640 1 -- 172.21.15.22:0/2090965100 >> 172.21.15.22:0/2090965100 conn(0x7faa740b7ea0 msgr2=0x7faa740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.986+0000 7faa7b340640 1 -- 172.21.15.22:0/2090965100 shutdown_connections 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.986+0000 7faa7b340640 1 -- 172.21.15.22:0/2090965100 wait complete. 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 10 monclient: build_initial_monmap 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 10 monclient: monmap: 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 1 librados: starting msgr at 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 1 librados: starting objecter 2024-03-25T18:20:23.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 1 Processor -- start 2024-03-25T18:20:23.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 1 -- start start 2024-03-25T18:20:23.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 1 librados: setting wanted keys 2024-03-25T18:20:23.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 1 librados: calling monclient init 2024-03-25T18:20:23.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.987+0000 7faa7b340640 10 monclient: init 2024-03-25T18:20:23.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 10 monclient(hunting): picked mon.noname-b con 0x7faa740a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 10 monclient(hunting): picked mon.noname-a con 0x7faa7413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 10 monclient(hunting): picked mon.noname-c con 0x7faa741430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740ab9c0 con 0x7faa7413f710 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa740abca0 con 0x7faa740a98f0 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa74137e80 con 0x7faa741430c0 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:23.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.988+0000 7faa7b340640 10 monclient(hunting): authenticate will time out at 7223.211145s 2024-03-25T18:20:23.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.995+0000 7faa788db640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7faa7413f710 0x7faa74133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44226/0 (socket says 172.21.15.22:44226) 2024-03-25T18:20:23.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.995+0000 7faa788db640 1 -- 172.21.15.22:0/3806224353 learned_addr learned my addr 172.21.15.22:0/3806224353 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:23.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.996+0000 7faa69ffb640 10 client.?.objecter ms_handle_connect 0x7faa741430c0 2024-03-25T18:20:23.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.996+0000 7faa69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:23.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.998+0000 7faa69ffb640 10 client.?.objecter ms_handle_connect 0x7faa7413f710 2024-03-25T18:20:23.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:23.998+0000 7faa69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.015+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373678481 0 0) 0x7faa740ab9c0 con 0x7faa7413f710 2024-03-25T18:20:24.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.015+0000 7faa69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.015+0000 7faa69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.015+0000 7faa69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.015+0000 7faa69ffb640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:20:24.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.015+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa58004eb0 con 0x7faa7413f710 2024-03-25T18:20:24.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.023+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458242185 0 0) 0x7faa74137e80 con 0x7faa741430c0 2024-03-25T18:20:24.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.023+0000 7faa69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.023+0000 7faa69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.023+0000 7faa69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.023+0000 7faa69ffb640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:20:24.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.023+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa740ab9c0 con 0x7faa741430c0 2024-03-25T18:20:24.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.037+0000 7faa69ffb640 10 client.?.objecter ms_handle_connect 0x7faa740a98f0 2024-03-25T18:20:24.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.037+0000 7faa69ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.040+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2575513717 0 0) 0x7faa58004eb0 con 0x7faa7413f710 2024-03-25T18:20:24.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.040+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa74137e80 con 0x7faa7413f710 2024-03-25T18:20:24.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.040+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa64001a10 con 0x7faa7413f710 2024-03-25T18:20:24.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.040+0000 7faa69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.040+0000 7faa69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:24.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.040+0000 7faa69ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:24.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.040+0000 7faa69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.042+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 910806757 0 0) 0x7faa740ab9c0 con 0x7faa741430c0 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.042+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa58004eb0 con 0x7faa741430c0 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.042+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa6c001b00 con 0x7faa741430c0 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.042+0000 7faa69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.042+0000 7faa69ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.042+0000 7faa69ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:24.042 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:24.043 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:24.043 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.043 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:24.043 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:24.043 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:24.043 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:24.043 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:24.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.048+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369180371 0 0) 0x7faa740abca0 con 0x7faa740a98f0 2024-03-25T18:20:24.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.049+0000 7faa69ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.049+0000 7faa69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.049+0000 7faa69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.049+0000 7faa69ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:20:24.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.049+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa740ab9c0 con 0x7faa740a98f0 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3460008164 0 0) 0x7faa58004eb0 con 0x7faa741430c0 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 >> v1:172.21.15.33:6789/0 conn(0x7faa740a98f0 legacy=0x7faa741d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 >> v1:172.21.15.22:6789/0 conn(0x7faa7413f710 legacy=0x7faa74133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 1 monclient: found mon.c 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:24.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa741d8150 con 0x7faa741430c0 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa7b340640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa7b340640 10 monclient: _renew_subs 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa7b340640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.056+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa74138eb0 con 0x7faa741430c0 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.057+0000 7faa7b340640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.057+0000 7faa7b340640 10 monclient: _renew_subs 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.057+0000 7faa7b340640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.057+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa74137b20 con 0x7faa741430c0 2024-03-25T18:20:24.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.057+0000 7faa7b340640 1 librados: init done 2024-03-25T18:20:24.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.058+0000 7faa7b340640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.058+0000 7faa7b340640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.058+0000 7faa7b340640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:24.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.058+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa740627b0 con 0x7faa741430c0 2024-03-25T18:20:24.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.061+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa6c001f30 con 0x7faa741430c0 2024-03-25T18:20:24.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.062+0000 7faa69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:24.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.062+0000 7faa6b7fe640 4 set_mon_vals no callback set 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.374+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa6c004d00 con 0x7faa741430c0 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.374+0000 7faa69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.374+0000 7faa69ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.374+0000 7faa69ffb640 10 monclient: dump: 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:24.375 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.376 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:24.376 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:24.376 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:24.376 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:24.376 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:24.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.375+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7faa6c004ee0 con 0x7faa741430c0 2024-03-25T18:20:24.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.376+0000 7faa790dc640 10 monclient: get_auth_request con 0x7faa5806df50 auth_method 0 2024-03-25T18:20:24.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.376+0000 7faa69ffb640 10 client.4274.objecter ms_handle_connect 0x7faa5806df50 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.383+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7faa6c054a20 con 0x7faa741430c0 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.383+0000 7faa69ffb640 10 client.4274.objecter ms_dispatch 0x7faa74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.383+0000 7faa69ffb640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.383+0000 7faa69ffb640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.384+0000 7faa69ffb640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.384+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa6c050530 con 0x7faa741430c0 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.384+0000 7faa69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.384+0000 7faa69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:24.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.644+0000 7faa7b340640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:24.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.644+0000 7faa7b340640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:24.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.644+0000 7faa7b340640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:24.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.644+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa740b71a0 con 0x7faa741430c0 2024-03-25T18:20:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.667+0000 7faa69ffb640 1 -- 172.21.15.22:0/3806224353 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7faa6c0058e0 con 0x7faa741430c0 2024-03-25T18:20:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.667+0000 7faa69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:24.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.667+0000 7faa69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:24.668 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:24.668 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:24.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.670+0000 7faa7b340640 10 librados: watch_flush enter 2024-03-25T18:20:24.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.670+0000 7faa7b340640 10 librados: watch_flush exit 2024-03-25T18:20:24.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.670+0000 7faa7b340640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-25T18:20:24.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.670+0000 7faa7b340640 10 client.4274.objecter successfully canceled tick 2024-03-25T18:20:24.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.670+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 >> v1:172.21.15.33:6800/215350958 conn(0x7faa5806df50 legacy=0x7faa58070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 10 monclient: shutdown 2024-03-25T18:20:24.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:24.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 >> v1:172.21.15.22:6790/0 conn(0x7faa741430c0 legacy=0x7faa741e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 shutdown_connections 2024-03-25T18:20:24.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 >> 172.21.15.22:0/3806224353 conn(0x7faa740b7ea0 msgr2=0x7faa74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:24.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 shutdown_connections 2024-03-25T18:20:24.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 1 -- 172.21.15.22:0/3806224353 wait complete. 2024-03-25T18:20:24.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:24.671+0000 7faa7b340640 1 librados: shutdown 2024-03-25T18:20:24.683 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:31.490 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.605+0000 7f7791f39640 10 monclient: get_monmap_and_config 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.605+0000 7f7791f39640 10 monclient: build_initial_monmap 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.605+0000 7f7791f39640 10 monclient: monmap: 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:31.606504+0000 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:31.606504+0000 2024-03-25T18:20:31.606 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:31.607 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:31.607 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:31.607 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:31.607 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:31.607 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:31.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.605+0000 7f7791f39640 1 Processor -- start 2024-03-25T18:20:31.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 1 -- start start 2024-03-25T18:20:31.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 10 monclient: init 2024-03-25T18:20:31.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:31.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:31.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 10 monclient(hunting): picked mon.noname-b con 0x7f778c0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:31.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 10 monclient(hunting): picked mon.noname-c con 0x7f778c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:31.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 10 monclient(hunting): picked mon.noname-a con 0x7f778c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:31.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.606+0000 7f7791f39640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c142520 con 0x7f778c1430c0 2024-03-25T18:20:31.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.607+0000 7f7791f39640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0b28a0 con 0x7f778c13f710 2024-03-25T18:20:31.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.607+0000 7f7791f39640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0b2a30 con 0x7f778c0a98f0 2024-03-25T18:20:31.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.607+0000 7f7791f39640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:31.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.607+0000 7f7791f39640 10 monclient(hunting): authenticate will time out at 7230.830688s 2024-03-25T18:20:31.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.614+0000 7f778bfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f778c1430c0 0x7f778c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48878/0 (socket says 172.21.15.22:48878) 2024-03-25T18:20:31.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.614+0000 7f778bfff640 1 -- 172.21.15.22:0/2852836427 learned_addr learned my addr 172.21.15.22:0/2852836427 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:31.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.632+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192325130 0 0) 0x7f778c142520 con 0x7f778c1430c0 2024-03-25T18:20:31.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.632+0000 7f778a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.632+0000 7f778a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.632+0000 7f778a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.632+0000 7f778a7fc640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:20:31.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.632+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774003520 con 0x7f778c1430c0 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.644+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094163136 0 0) 0x7f778c0b2a30 con 0x7f778c0a98f0 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.644+0000 7f778a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.644+0000 7f778a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.644+0000 7f778a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.644+0000 7f778a7fc640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:20:31.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.644+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778c142520 con 0x7f778c0a98f0 2024-03-25T18:20:31.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.649+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646002096 0 0) 0x7f778c0b28a0 con 0x7f778c13f710 2024-03-25T18:20:31.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.649+0000 7f778a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.649+0000 7f778a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.649+0000 7f778a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.649+0000 7f778a7fc640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:20:31.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.649+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778c0b2a30 con 0x7f778c13f710 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1989355308 0 0) 0x7f7774003520 con 0x7f778c1430c0 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 >> v1:172.21.15.22:6790/0 conn(0x7f778c0a98f0 legacy=0x7f778c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 >> v1:172.21.15.33:6789/0 conn(0x7f778c13f710 legacy=0x7f778c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0b44c0 con 0x7f778c1430c0 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:31.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7784001a10 con 0x7f778c1430c0 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f778a7fc640 10 monclient: dump: 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f7791f39640 5 monclient: authenticate success, global_id 4275 2024-03-25T18:20:31.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.650+0000 7f7791f39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7784001b70 con 0x7f778c1430c0 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f778a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f778a7fc640 1 -- 172.21.15.22:0/2852836427 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7784001f80 con 0x7f778c1430c0 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f778a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f778a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f778a7fc640 10 monclient: dump: 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:31.658 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f7791f39640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:31.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.656+0000 7f7791f39640 4 set_mon_vals no callback set 2024-03-25T18:20:31.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.657+0000 7f7791f39640 10 monclient: shutdown 2024-03-25T18:20:31.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.657+0000 7f7791f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:31.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.657+0000 7f7791f39640 1 -- 172.21.15.22:0/2852836427 >> v1:172.21.15.22:6789/0 conn(0x7f778c1430c0 legacy=0x7f778c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.657+0000 7f7791f39640 1 -- 172.21.15.22:0/2852836427 shutdown_connections 2024-03-25T18:20:31.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.657+0000 7f7791f39640 1 -- 172.21.15.22:0/2852836427 >> 172.21.15.22:0/2852836427 conn(0x7f778c0b7ea0 msgr2=0x7f778c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:31.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.657+0000 7f7791f39640 1 -- 172.21.15.22:0/2852836427 shutdown_connections 2024-03-25T18:20:31.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.657+0000 7f7791f39640 1 -- 172.21.15.22:0/2852836427 wait complete. 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.658+0000 7f7791f39640 10 monclient: build_initial_monmap 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.658+0000 7f7791f39640 10 monclient: monmap: 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.658+0000 7f7791f39640 1 librados: starting msgr at 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.658+0000 7f7791f39640 1 librados: starting objecter 2024-03-25T18:20:31.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.658+0000 7f7791f39640 1 Processor -- start 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 1 -- start start 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 1 librados: setting wanted keys 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 1 librados: calling monclient init 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient: init 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient(hunting): picked mon.noname-c con 0x7f778c0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient(hunting): picked mon.noname-a con 0x7f778c13f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient(hunting): picked mon.noname-b con 0x7f778c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c139830 con 0x7f778c13f710 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c1db7d0 con 0x7f778c1430c0 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c1db410 con 0x7f778c0a98f0 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:31.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.659+0000 7f7791f39640 10 monclient(hunting): authenticate will time out at 7230.883239s 2024-03-25T18:20:31.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.673+0000 7f778affd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f778c13f710 0x7f778c0b0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48940/0 (socket says 172.21.15.22:48940) 2024-03-25T18:20:31.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.673+0000 7f778affd640 1 -- 172.21.15.22:0/2178810891 learned_addr learned my addr 172.21.15.22:0/2178810891 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:31.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.674+0000 7f7773fff640 10 client.?.objecter ms_handle_connect 0x7f778c13f710 2024-03-25T18:20:31.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.674+0000 7f7773fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:31.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.682+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622807055 0 0) 0x7f778c139830 con 0x7f778c13f710 2024-03-25T18:20:31.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.682+0000 7f7773fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.682+0000 7f7773fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.682+0000 7f7773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.682+0000 7f7773fff640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:20:31.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.682+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7778004eb0 con 0x7f778c13f710 2024-03-25T18:20:31.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.687+0000 7f7773fff640 10 client.?.objecter ms_handle_connect 0x7f778c1430c0 2024-03-25T18:20:31.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.687+0000 7f7773fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:31.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.690+0000 7f7773fff640 10 client.?.objecter ms_handle_connect 0x7f778c0a98f0 2024-03-25T18:20:31.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.690+0000 7f7773fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:31.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.697+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484226321 0 0) 0x7f778c1db7d0 con 0x7f778c1430c0 2024-03-25T18:20:31.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.697+0000 7f7773fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.697+0000 7f7773fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.697+0000 7f7773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.697+0000 7f7773fff640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:20:31.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.697+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778c139830 con 0x7f778c1430c0 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.705+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010739304 0 0) 0x7f778c1db410 con 0x7f778c0a98f0 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.705+0000 7f7773fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.705+0000 7f7773fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.705+0000 7f7773fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.705+0000 7f7773fff640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.705+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778c1db7d0 con 0x7f778c0a98f0 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.706+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 853389244 0 0) 0x7f7778004eb0 con 0x7f778c13f710 2024-03-25T18:20:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.706+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f778c1db410 con 0x7f778c13f710 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.706+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7780001a10 con 0x7f778c13f710 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.706+0000 7f7773fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.706+0000 7f7773fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.706+0000 7f7773fff640 10 monclient(hunting): dump: 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:31.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.706+0000 7f7773fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:31.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 37393503 0 0) 0x7f778c1db410 con 0x7f778c13f710 2024-03-25T18:20:31.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 >> v1:172.21.15.22:6790/0 conn(0x7f778c0a98f0 legacy=0x7f778c1dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 >> v1:172.21.15.33:6789/0 conn(0x7f778c1430c0 legacy=0x7f778c1d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 1 monclient: found mon.a 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c1db5a0 con 0x7f778c13f710 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7773fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 10 monclient: _renew_subs 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c0b1480 con 0x7f778c13f710 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:31.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 10 monclient: _renew_subs 2024-03-25T18:20:31.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:31.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c0b0b10 con 0x7f778c13f710 2024-03-25T18:20:31.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.716+0000 7f7791f39640 1 librados: init done 2024-03-25T18:20:31.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.717+0000 7f7791f39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:31.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.717+0000 7f7791f39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:31.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.717+0000 7f7791f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:31.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.717+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c1db960 con 0x7f778c13f710 2024-03-25T18:20:31.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.727+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7780001c70 con 0x7f778c13f710 2024-03-25T18:20:31.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.727+0000 7f7773fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:31.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.727+0000 7f7789ffb640 4 set_mon_vals no callback set 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.730+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7780003a60 con 0x7f778c13f710 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.730+0000 7f7773fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.730+0000 7f7773fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.730+0000 7f7773fff640 10 monclient: dump: 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.731 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:31.732 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:31.732 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:31.732 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:31.732 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:31.732 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:31.732 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:31.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.993+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f7780003c40 con 0x7f778c13f710 2024-03-25T18:20:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.994+0000 7f778b7fe640 10 monclient: get_auth_request con 0x7f777806ddf0 auth_method 0 2024-03-25T18:20:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:31.994+0000 7f7773fff640 10 client.4278.objecter ms_handle_connect 0x7f777806ddf0 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f77800548d0 con 0x7f778c13f710 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 10 client.4278.objecter ms_dispatch 0x7f778c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7780050210 con 0x7f778c13f710 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.012+0000 7f7773fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:32.164 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:34.827 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.216+0000 7f7791f39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:32.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.216+0000 7f7791f39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:32.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.216+0000 7f7791f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:32.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.216+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f778c0627b0 con 0x7f778c13f710 2024-03-25T18:20:32.220 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:34.882 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.287 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:34.949 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.288+0000 7f7773fff640 1 -- 172.21.15.22:0/2178810891 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f7780052310 con 0x7f778c13f710 2024-03-25T18:20:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.288+0000 7f7773fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.288+0000 7f7773fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:32.290 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:32.290 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.290+0000 7f7791f39640 10 librados: watch_flush enter 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.290+0000 7f7791f39640 10 librados: watch_flush exit 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.290+0000 7f7791f39640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.290+0000 7f7791f39640 10 client.4278.objecter successfully canceled tick 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 >> v1:172.21.15.33:6800/215350958 conn(0x7f777806ddf0 legacy=0x7f77780702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 10 monclient: shutdown 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 >> v1:172.21.15.22:6789/0 conn(0x7f778c13f710 legacy=0x7f778c0b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 shutdown_connections 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 >> 172.21.15.22:0/2178810891 conn(0x7f778c0b7ea0 msgr2=0x7f778c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 shutdown_connections 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 1 -- 172.21.15.22:0/2178810891 wait complete. 2024-03-25T18:20:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:32.291+0000 7f7791f39640 1 librados: shutdown 2024-03-25T18:20:32.294 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:34.910 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.294 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:34.956 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.300 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:34.918 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.301 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:34.963 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:32.346 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:34.964 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:32.364 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:35.026 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:39.109 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:39.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.224+0000 7f4b62923640 10 monclient: get_monmap_and_config 2024-03-25T18:20:39.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.224+0000 7f4b62923640 10 monclient: build_initial_monmap 2024-03-25T18:20:39.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.224+0000 7f4b62923640 10 monclient: monmap: 2024-03-25T18:20:39.225 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:39.225861+0000 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:39.225861+0000 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:39.226 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:39.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.225+0000 7f4b62923640 1 Processor -- start 2024-03-25T18:20:39.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.225+0000 7f4b62923640 1 -- start start 2024-03-25T18:20:39.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.225+0000 7f4b62923640 10 monclient: init 2024-03-25T18:20:39.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:39.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 10 monclient(hunting): picked mon.noname-a con 0x7f4b5c0b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 10 monclient(hunting): picked mon.noname-b con 0x7f4b5c0b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 10 monclient(hunting): picked mon.noname-c con 0x7f4b5c14be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c0af940 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c0b0a70 con 0x7f4b5c14be90 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c0b0d50 con 0x7f4b5c0b1730 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:39.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.226+0000 7f4b62923640 10 monclient(hunting): authenticate will time out at 7238.449989s 2024-03-25T18:20:39.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.233+0000 7f4b60ec0640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f4b5c14be90 0x7f4b5c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:44318/0 (socket says 172.21.15.22:44318) 2024-03-25T18:20:39.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.233+0000 7f4b60ec0640 1 -- 172.21.15.22:0/1240746778 learned_addr learned my addr 172.21.15.22:0/1240746778 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:39.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.240+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732029968 0 0) 0x7f4b5c0b0d50 con 0x7f4b5c0b1730 2024-03-25T18:20:39.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.241+0000 7f4b5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:39.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.241+0000 7f4b5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:39.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.241+0000 7f4b5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:39.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.241+0000 7f4b5affd640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:20:39.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.241+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b44003520 con 0x7f4b5c0b1730 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553727686 0 0) 0x7f4b44003520 con 0x7f4b5c0b1730 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 >> v1:172.21.15.33:6789/0 conn(0x7f4b5c14be90 legacy=0x7f4b5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 >> v1:172.21.15.22:6789/0 conn(0x7f4b5c0b4d60 legacy=0x7f4b5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 1 monclient: found mon.noname-b 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c0b1030 con 0x7f4b5c0b1730 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 10 monclient: _finish_auth 0 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4b50001690 con 0x7f4b5c0b1730 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b62923640 5 monclient: authenticate success, global_id 4283 2024-03-25T18:20:39.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b62923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b5affd640 10 monclient: dump: 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:39.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.250+0000 7f4b62923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:39.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b50001860 con 0x7f4b5c0b1730 2024-03-25T18:20:39.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:39.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b5affd640 1 -- 172.21.15.22:0/1240746778 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4b50001cb0 con 0x7f4b5c0b1730 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b62923640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b62923640 4 set_mon_vals no callback set 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.269+0000 7f4b5affd640 10 monclient: dump: 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:39.271 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.270+0000 7f4b62923640 10 monclient: shutdown 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.270+0000 7f4b62923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.270+0000 7f4b62923640 1 -- 172.21.15.22:0/1240746778 >> v1:172.21.15.22:6790/0 conn(0x7f4b5c0b1730 legacy=0x7f4b5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.270+0000 7f4b62923640 1 -- 172.21.15.22:0/1240746778 shutdown_connections 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.270+0000 7f4b62923640 1 -- 172.21.15.22:0/1240746778 >> 172.21.15.22:0/1240746778 conn(0x7f4b5c0617a0 msgr2=0x7f4b5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.270+0000 7f4b62923640 1 -- 172.21.15.22:0/1240746778 shutdown_connections 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.270+0000 7f4b62923640 1 -- 172.21.15.22:0/1240746778 wait complete. 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.271+0000 7f4b62923640 10 monclient: build_initial_monmap 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.271+0000 7f4b62923640 10 monclient: monmap: 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:39.273 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.271+0000 7f4b62923640 1 librados: starting msgr at 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.271+0000 7f4b62923640 1 librados: starting objecter 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.271+0000 7f4b62923640 1 Processor -- start 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 1 -- start start 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 1 librados: setting wanted keys 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 1 librados: calling monclient init 2024-03-25T18:20:39.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient: init 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient(hunting): picked mon.noname-a con 0x7f4b5c0b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient(hunting): picked mon.noname-c con 0x7f4b5c0b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient(hunting): picked mon.noname-b con 0x7f4b5c14be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c137d00 con 0x7f4b5c0b1730 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c140d60 con 0x7f4b5c14be90 2024-03-25T18:20:39.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b5c140ef0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:39.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.272+0000 7f4b62923640 10 monclient(hunting): authenticate will time out at 7238.496243s 2024-03-25T18:20:39.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.281+0000 7f4b5b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4b5c0b4d60 0x7f4b5c142880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:36916/0 (socket says 172.21.15.22:36916) 2024-03-25T18:20:39.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.281+0000 7f4b5b7fe640 1 -- 172.21.15.22:0/804586660 learned_addr learned my addr 172.21.15.22:0/804586660 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:39.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.286+0000 7f4b58ff9640 10 client.?.objecter ms_handle_connect 0x7f4b5c0b4d60 2024-03-25T18:20:39.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.286+0000 7f4b58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.295+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903820528 0 0) 0x7f4b5c140ef0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.295+0000 7f4b58ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.295+0000 7f4b58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.295+0000 7f4b58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.295+0000 7f4b58ff9640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:20:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.295+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b48004eb0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.299+0000 7f4b58ff9640 10 client.?.objecter ms_handle_connect 0x7f4b5c0b1730 2024-03-25T18:20:39.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.299+0000 7f4b58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.302+0000 7f4b58ff9640 10 client.?.objecter ms_handle_connect 0x7f4b5c14be90 2024-03-25T18:20:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.302+0000 7f4b58ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:39.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.303+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2280127478 0 0) 0x7f4b48004eb0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.303+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b5c140ef0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.303+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4b50001bb0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.303+0000 7f4b58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.303+0000 7f4b58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.303+0000 7f4b58ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:39.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.303+0000 7f4b58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 945749361 0 0) 0x7f4b5c140ef0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 >> v1:172.21.15.33:6789/0 conn(0x7f4b5c14be90 legacy=0x7f4b5c1e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 >> v1:172.21.15.22:6789/0 conn(0x7f4b5c0b1730 legacy=0x7f4b5c136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 1 monclient: found mon.c 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c1382e0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 10 monclient: _renew_subs 2024-03-25T18:20:39.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b5c1384d0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 10 monclient: _renew_subs 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b5c140560 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.310+0000 7f4b62923640 1 librados: init done 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.312+0000 7f4b62923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:39.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.312+0000 7f4b62923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:39.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.312+0000 7f4b62923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:39.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.312+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b5c0b1bc0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.526+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b50003960 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.526+0000 7f4b58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:39.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.526+0000 7f4b5a7fc640 4 set_mon_vals no callback set 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.631+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4b50001270 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.631+0000 7f4b58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.631+0000 7f4b58ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.631+0000 7f4b58ff9640 10 monclient: dump: 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:39.632 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:39.633 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:39.633 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:39.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.631+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f4b50003e30 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.632+0000 7f4b5bfff640 10 monclient: get_auth_request con 0x7f4b4806dce0 auth_method 0 2024-03-25T18:20:39.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.632+0000 7f4b58ff9640 10 client.4286.objecter ms_handle_connect 0x7f4b4806dce0 2024-03-25T18:20:39.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f4b50054950 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 10 client.4286.objecter ms_dispatch 0x7f4b5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-25T18:20:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b50050460 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.638+0000 7f4b58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:39.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.897+0000 7f4b62923640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:39.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.897+0000 7f4b62923640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:39.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.897+0000 7f4b62923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:39.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.897+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b5c1401f0 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.920+0000 7f4b58ff9640 1 -- 172.21.15.22:0/804586660 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f4b50052390 con 0x7f4b5c0b4d60 2024-03-25T18:20:39.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.920+0000 7f4b58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:39.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.920+0000 7f4b58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:39.922 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:39.922 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:39.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.922+0000 7f4b62923640 10 librados: watch_flush enter 2024-03-25T18:20:39.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.922+0000 7f4b62923640 10 librados: watch_flush exit 2024-03-25T18:20:39.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.922+0000 7f4b62923640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-25T18:20:39.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.922+0000 7f4b62923640 10 client.4286.objecter successfully canceled tick 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 >> v1:172.21.15.33:6800/215350958 conn(0x7f4b4806dce0 legacy=0x7f4b480701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 10 monclient: shutdown 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 >> v1:172.21.15.22:6790/0 conn(0x7f4b5c0b4d60 legacy=0x7f4b5c142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 shutdown_connections 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 >> 172.21.15.22:0/804586660 conn(0x7f4b5c0617a0 msgr2=0x7f4b5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 shutdown_connections 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 1 -- 172.21.15.22:0/804586660 wait complete. 2024-03-25T18:20:39.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:39.923+0000 7f4b62923640 1 librados: shutdown 2024-03-25T18:20:39.936 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:46.743 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:46.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.854+0000 7fa483715640 10 monclient: get_monmap_and_config 2024-03-25T18:20:46.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.854+0000 7fa483715640 10 monclient: build_initial_monmap 2024-03-25T18:20:46.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.854+0000 7fa483715640 10 monclient: monmap: 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:46.855522+0000 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:46.855522+0000 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:46.856 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:46.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.855+0000 7fa483715640 1 Processor -- start 2024-03-25T18:20:46.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.855+0000 7fa483715640 1 -- start start 2024-03-25T18:20:46.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.855+0000 7fa483715640 10 monclient: init 2024-03-25T18:20:46.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.855+0000 7fa483715640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:46.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.855+0000 7fa483715640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:46.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.855+0000 7fa483715640 10 monclient(hunting): picked mon.noname-b con 0x7fa47c0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:46.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.856+0000 7fa483715640 10 monclient(hunting): picked mon.noname-a con 0x7fa47c13f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:46.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.856+0000 7fa483715640 10 monclient(hunting): picked mon.noname-c con 0x7fa47c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:46.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.856+0000 7fa483715640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c142520 con 0x7fa47c13f710 2024-03-25T18:20:46.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.856+0000 7fa483715640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c0b28a0 con 0x7fa47c1430c0 2024-03-25T18:20:46.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.856+0000 7fa483715640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c0b2a30 con 0x7fa47c0a98f0 2024-03-25T18:20:46.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.856+0000 7fa483715640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:46.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.856+0000 7fa483715640 10 monclient(hunting): authenticate will time out at 7246.079631s 2024-03-25T18:20:46.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.872+0000 7fa480cb0640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa47c13f710 0x7fa47c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:49894/0 (socket says 172.21.15.22:49894) 2024-03-25T18:20:46.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.872+0000 7fa480cb0640 1 -- 172.21.15.22:0/3650471691 learned_addr learned my addr 172.21.15.22:0/3650471691 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:46.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.884+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148321459 0 0) 0x7fa47c142520 con 0x7fa47c13f710 2024-03-25T18:20:46.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.884+0000 7fa473fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.884+0000 7fa473fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.884+0000 7fa473fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.884+0000 7fa473fff640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:20:46.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.885+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa468003520 con 0x7fa47c13f710 2024-03-25T18:20:46.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.885+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4126021479 0 0) 0x7fa47c0b28a0 con 0x7fa47c1430c0 2024-03-25T18:20:46.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.885+0000 7fa473fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.885+0000 7fa473fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.885+0000 7fa473fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.885+0000 7fa473fff640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:20:46.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.885+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa47c142520 con 0x7fa47c1430c0 2024-03-25T18:20:46.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 299927607 0 0) 0x7fa468003520 con 0x7fa47c13f710 2024-03-25T18:20:46.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 >> v1:172.21.15.22:6790/0 conn(0x7fa47c0a98f0 legacy=0x7fa47c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 >> v1:172.21.15.33:6789/0 conn(0x7fa47c1430c0 legacy=0x7fa47c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:46.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 1 monclient: found mon.noname-a 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c0b44c0 con 0x7fa47c13f710 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa46c001690 con 0x7fa47c13f710 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa473fff640 10 monclient: dump: 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:46.896 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:46.897 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:46.897 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:46.897 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:46.897 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:46.897 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:46.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa483715640 5 monclient: authenticate success, global_id 4284 2024-03-25T18:20:46.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.894+0000 7fa483715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa46c0017f0 con 0x7fa47c13f710 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa473fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa473fff640 1 -- 172.21.15.22:0/3650471691 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa46c001c00 con 0x7fa47c13f710 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa483715640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa483715640 4 set_mon_vals no callback set 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa473fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa473fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa473fff640 10 monclient: dump: 2024-03-25T18:20:46.910 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:46.911 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:46.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa483715640 10 monclient: shutdown 2024-03-25T18:20:46.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa483715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.909+0000 7fa483715640 1 -- 172.21.15.22:0/3650471691 >> v1:172.21.15.22:6789/0 conn(0x7fa47c13f710 legacy=0x7fa47c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.910+0000 7fa483715640 1 -- 172.21.15.22:0/3650471691 shutdown_connections 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.910+0000 7fa483715640 1 -- 172.21.15.22:0/3650471691 >> 172.21.15.22:0/3650471691 conn(0x7fa47c0b7ea0 msgr2=0x7fa47c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.910+0000 7fa483715640 1 -- 172.21.15.22:0/3650471691 shutdown_connections 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.910+0000 7fa483715640 1 -- 172.21.15.22:0/3650471691 wait complete. 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.910+0000 7fa483715640 10 monclient: build_initial_monmap 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.910+0000 7fa483715640 10 monclient: monmap: 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:46.913 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.911+0000 7fa483715640 1 librados: starting msgr at 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.911+0000 7fa483715640 1 librados: starting objecter 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.911+0000 7fa483715640 1 Processor -- start 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.911+0000 7fa483715640 1 -- start start 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.911+0000 7fa483715640 1 librados: setting wanted keys 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.911+0000 7fa483715640 1 librados: calling monclient init 2024-03-25T18:20:46.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.911+0000 7fa483715640 10 monclient: init 2024-03-25T18:20:46.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:46.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:46.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 10 monclient(hunting): picked mon.noname-a con 0x7fa47c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:46.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 10 monclient(hunting): picked mon.noname-c con 0x7fa47c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:46.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 10 monclient(hunting): picked mon.noname-b con 0x7fa47c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:46.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c139520 con 0x7fa47c0a98f0 2024-03-25T18:20:46.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c138b40 con 0x7fa47c1430c0 2024-03-25T18:20:46.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c138e20 con 0x7fa47c13f710 2024-03-25T18:20:46.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:46.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.912+0000 7fa483715640 10 monclient(hunting): authenticate will time out at 7246.135814s 2024-03-25T18:20:46.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.927+0000 7fa480cb0640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa47c13f710 0x7fa47c137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40276/0 (socket says 172.21.15.22:40276) 2024-03-25T18:20:46.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.927+0000 7fa480cb0640 1 -- 172.21.15.22:0/1003085602 learned_addr learned my addr 172.21.15.22:0/1003085602 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:46.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.931+0000 7fa471ffb640 10 client.?.objecter ms_handle_connect 0x7fa47c13f710 2024-03-25T18:20:46.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.931+0000 7fa471ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:46.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.938+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242365510 0 0) 0x7fa47c138e20 con 0x7fa47c13f710 2024-03-25T18:20:46.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.939+0000 7fa471ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.939+0000 7fa471ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.939+0000 7fa471ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.939+0000 7fa471ffb640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:20:46.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.939+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa460004eb0 con 0x7fa47c13f710 2024-03-25T18:20:46.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.940+0000 7fa471ffb640 10 client.?.objecter ms_handle_connect 0x7fa47c0a98f0 2024-03-25T18:20:46.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.940+0000 7fa471ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:46.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.943+0000 7fa471ffb640 10 client.?.objecter ms_handle_connect 0x7fa47c1430c0 2024-03-25T18:20:46.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.943+0000 7fa471ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:46.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.948+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799635362 0 0) 0x7fa47c138b40 con 0x7fa47c1430c0 2024-03-25T18:20:46.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.948+0000 7fa471ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.948+0000 7fa471ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.948+0000 7fa471ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.948+0000 7fa471ffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:20:46.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.948+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa47c138e20 con 0x7fa47c1430c0 2024-03-25T18:20:46.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.953+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007352791 0 0) 0x7fa460004eb0 con 0x7fa47c13f710 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.953+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa47c138b40 con 0x7fa47c13f710 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.953+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa46c001bc0 con 0x7fa47c13f710 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.953+0000 7fa471ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.953+0000 7fa471ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.953+0000 7fa471ffb640 10 monclient(hunting): dump: 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:46.955 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.953+0000 7fa471ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.954+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877025933 0 0) 0x7fa47c139520 con 0x7fa47c0a98f0 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.954+0000 7fa471ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.954+0000 7fa471ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.954+0000 7fa471ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.954+0000 7fa471ffb640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:20:46.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.954+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa460004eb0 con 0x7fa47c0a98f0 2024-03-25T18:20:46.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2473692398 0 0) 0x7fa47c138b40 con 0x7fa47c13f710 2024-03-25T18:20:46.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 >> v1:172.21.15.33:6789/0 conn(0x7fa47c1430c0 legacy=0x7fa47c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 >> v1:172.21.15.22:6789/0 conn(0x7fa47c0a98f0 legacy=0x7fa47c137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 1 monclient: found mon.c 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c139100 con 0x7fa47c13f710 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa471ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa483715640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa483715640 10 monclient: _renew_subs 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa483715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.960+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47c136aa0 con 0x7fa47c13f710 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.961+0000 7fa483715640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:46.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.961+0000 7fa483715640 10 monclient: _renew_subs 2024-03-25T18:20:46.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.961+0000 7fa483715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:46.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.961+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c13ba10 con 0x7fa47c13f710 2024-03-25T18:20:46.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.961+0000 7fa483715640 1 librados: init done 2024-03-25T18:20:46.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.962+0000 7fa483715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:46.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.962+0000 7fa483715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:46.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.962+0000 7fa483715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:46.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:46.962+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47c136c90 con 0x7fa47c13f710 2024-03-25T18:20:47.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.286+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa46c0011f0 con 0x7fa47c13f710 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.287+0000 7fa471ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.287+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa46c004c30 con 0x7fa47c13f710 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.287+0000 7fa471ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.287+0000 7fa471ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.287+0000 7fa471ffb640 10 monclient: dump: 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:47.289 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:47.290 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:47.290 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:47.290 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:47.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.287+0000 7fa4737fe640 4 set_mon_vals no callback set 2024-03-25T18:20:47.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.287+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fa46c01ac80 con 0x7fa47c13f710 2024-03-25T18:20:47.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.288+0000 7fa4814b1640 10 monclient: get_auth_request con 0x7fa46006ddb0 auth_method 0 2024-03-25T18:20:47.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.288+0000 7fa471ffb640 10 client.4292.objecter ms_handle_connect 0x7fa46006ddb0 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fa46c054970 con 0x7fa47c13f710 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 10 client.4292.objecter ms_dispatch 0x7fa47c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa46c0502b0 con 0x7fa47c13f710 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:47.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.293+0000 7fa471ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:47.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.562+0000 7fa483715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:47.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.562+0000 7fa483715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:47.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.562+0000 7fa483715640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:20:47.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.562+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa47c0627b0 con 0x7fa47c13f710 2024-03-25T18:20:47.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.581+0000 7fa471ffb640 1 -- 172.21.15.22:0/1003085602 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fa46c0523b0 con 0x7fa47c13f710 2024-03-25T18:20:47.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.581+0000 7fa471ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:47.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.581+0000 7fa471ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:47.583 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:47.583 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 10 librados: watch_flush enter 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 10 librados: watch_flush exit 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 10 client.4292.objecter successfully canceled tick 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 >> v1:172.21.15.33:6800/215350958 conn(0x7fa46006ddb0 legacy=0x7fa460070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 10 monclient: shutdown 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 >> v1:172.21.15.22:6790/0 conn(0x7fa47c13f710 legacy=0x7fa47c137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 shutdown_connections 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 >> 172.21.15.22:0/1003085602 conn(0x7fa47c0b7ea0 msgr2=0x7fa47c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 shutdown_connections 2024-03-25T18:20:47.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 1 -- 172.21.15.22:0/1003085602 wait complete. 2024-03-25T18:20:47.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:47.584+0000 7fa483715640 1 librados: shutdown 2024-03-25T18:20:47.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:52.155 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:54.817 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.169 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:54.831 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.200 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:01:54.862 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.213 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:54.831 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.220 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:54.838 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.227 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:54.890 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.239 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:54.901 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.253 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:01:54.872 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:52.279 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:01:54.941 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.403 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:54.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.516+0000 7fc208a1e640 10 monclient: get_monmap_and_config 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.516+0000 7fc208a1e640 10 monclient: build_initial_monmap 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.516+0000 7fc208a1e640 10 monclient: monmap: 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:20:54.517899+0000 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:20:54.517899+0000 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:20:54.518 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.517+0000 7fc208a1e640 1 Processor -- start 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.517+0000 7fc208a1e640 1 -- start start 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.517+0000 7fc208a1e640 10 monclient: init 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2040a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc20413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc2041430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc204142520 con 0x7fc2041430c0 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2040b28a0 con 0x7fc20413f710 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2040b2a30 con 0x7fc2040a98f0 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:54.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.518+0000 7fc208a1e640 10 monclient(hunting): authenticate will time out at 7253.741813s 2024-03-25T18:20:54.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.526+0000 7fc1fbfff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fc20413f710 0x7fc2041429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:53108/0 (socket says 172.21.15.22:53108) 2024-03-25T18:20:54.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.526+0000 7fc1fbfff640 1 -- 172.21.15.22:0/748628613 learned_addr learned my addr 172.21.15.22:0/748628613 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:54.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.537+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194521766 0 0) 0x7fc2040b28a0 con 0x7fc20413f710 2024-03-25T18:20:54.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.537+0000 7fc201d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:54.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.537+0000 7fc201d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:54.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.537+0000 7fc201d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:54.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.537+0000 7fc201d9b640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:20:54.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.537+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f0003520 con 0x7fc20413f710 2024-03-25T18:20:54.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.545+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013063341 0 0) 0x7fc2040b2a30 con 0x7fc2040a98f0 2024-03-25T18:20:54.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.545+0000 7fc201d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:54.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.545+0000 7fc201d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:54.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.545+0000 7fc201d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:54.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.545+0000 7fc201d9b640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:20:54.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.545+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2040b28a0 con 0x7fc2040a98f0 2024-03-25T18:20:54.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309369629 0 0) 0x7fc1f0003520 con 0x7fc20413f710 2024-03-25T18:20:54.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 >> v1:172.21.15.22:6790/0 conn(0x7fc2040a98f0 legacy=0x7fc2040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:54.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 >> v1:172.21.15.22:6789/0 conn(0x7fc2041430c0 legacy=0x7fc20414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:54.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:54.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 1 monclient: found mon.noname-c 2024-03-25T18:20:54.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:54.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2040b44c0 con 0x7fc20413f710 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc1ec001690 con 0x7fc20413f710 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc208a1e640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc208a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc201d9b640 10 monclient: dump: 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:54.561 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:54.562 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:54.562 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:54.562 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.559+0000 7fc208a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1ec0017f0 con 0x7fc20413f710 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc201d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc201d9b640 1 -- 172.21.15.22:0/748628613 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc1ec001c00 con 0x7fc20413f710 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc208a1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc208a1e640 4 set_mon_vals no callback set 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc201d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc201d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc201d9b640 10 monclient: dump: 2024-03-25T18:20:54.590 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:54.591 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc208a1e640 10 monclient: shutdown 2024-03-25T18:20:54.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc208a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:54.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.589+0000 7fc208a1e640 1 -- 172.21.15.22:0/748628613 >> v1:172.21.15.33:6789/0 conn(0x7fc20413f710 legacy=0x7fc2041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:54.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.590+0000 7fc208a1e640 1 -- 172.21.15.22:0/748628613 shutdown_connections 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.590+0000 7fc208a1e640 1 -- 172.21.15.22:0/748628613 >> 172.21.15.22:0/748628613 conn(0x7fc2040b7ea0 msgr2=0x7fc2040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.590+0000 7fc208a1e640 1 -- 172.21.15.22:0/748628613 shutdown_connections 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.590+0000 7fc208a1e640 1 -- 172.21.15.22:0/748628613 wait complete. 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 10 monclient: build_initial_monmap 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 10 monclient: monmap: 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 1 librados: starting msgr at 2024-03-25T18:20:54.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 1 librados: starting objecter 2024-03-25T18:20:54.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 1 Processor -- start 2024-03-25T18:20:54.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 1 -- start start 2024-03-25T18:20:54.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 1 librados: setting wanted keys 2024-03-25T18:20:54.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 1 librados: calling monclient init 2024-03-25T18:20:54.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.591+0000 7fc208a1e640 10 monclient: init 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fc2040a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fc20413f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fc2041430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc204139a40 con 0x7fc2040a98f0 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc204133f20 con 0x7fc2041430c0 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2040ab950 con 0x7fc20413f710 2024-03-25T18:20:54.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:54.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.592+0000 7fc208a1e640 10 monclient(hunting): authenticate will time out at 7253.816017s 2024-03-25T18:20:54.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.600+0000 7fc202d9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fc2041430c0 0x7fc204133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:53158/0 (socket says 172.21.15.22:53158) 2024-03-25T18:20:54.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.600+0000 7fc202d9d640 1 -- 172.21.15.22:0/3285301964 learned_addr learned my addr 172.21.15.22:0/3285301964 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:20:54.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.604+0000 7fc1faffd640 10 client.?.objecter ms_handle_connect 0x7fc2041430c0 2024-03-25T18:20:54.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.604+0000 7fc1faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:54.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.612+0000 7fc1faffd640 10 client.?.objecter ms_handle_connect 0x7fc20413f710 2024-03-25T18:20:54.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.612+0000 7fc1faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:54.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.613+0000 7fc1faffd640 10 client.?.objecter ms_handle_connect 0x7fc2040a98f0 2024-03-25T18:20:54.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.613+0000 7fc1faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.618+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227346054 0 0) 0x7fc2040ab950 con 0x7fc20413f710 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.618+0000 7fc1faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.618+0000 7fc1faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.618+0000 7fc1faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.618+0000 7fc1faffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.618+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e4004eb0 con 0x7fc20413f710 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.619+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228828520 0 0) 0x7fc204133f20 con 0x7fc2041430c0 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.619+0000 7fc1faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.619+0000 7fc1faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.619+0000 7fc1faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.619+0000 7fc1faffd640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:20:54.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.619+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1e40051d0 con 0x7fc2041430c0 2024-03-25T18:20:54.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.627+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915390760 0 0) 0x7fc204139a40 con 0x7fc2040a98f0 2024-03-25T18:20:54.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.627+0000 7fc1faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:54.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.627+0000 7fc1faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:54.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.627+0000 7fc1faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:54.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.627+0000 7fc1faffd640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:20:54.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.627+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc204133f20 con 0x7fc2040a98f0 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.633+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1887011944 0 0) 0x7fc1e40051d0 con 0x7fc2041430c0 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.633+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc204139a40 con 0x7fc2041430c0 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.633+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc1fc001c10 con 0x7fc2041430c0 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.633+0000 7fc1faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.633+0000 7fc1faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.633+0000 7fc1faffd640 10 monclient(hunting): dump: 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:54.634 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.633+0000 7fc1faffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.635+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2676655195 0 0) 0x7fc204133f20 con 0x7fc2040a98f0 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.635+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1e40051d0 con 0x7fc2040a98f0 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.635+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc1f40018f0 con 0x7fc2040a98f0 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.635+0000 7fc1faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.635+0000 7fc1faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.635+0000 7fc1faffd640 10 monclient(hunting): dump: 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.636 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.637 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:54.637 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:54.637 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:54.637 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:54.637 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:54.637 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.645+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945873701 0 0) 0x7fc1e4004eb0 con 0x7fc20413f710 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.645+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc204133f20 con 0x7fc20413f710 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.645+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc1ec001b00 con 0x7fc20413f710 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.645+0000 7fc1faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.645+0000 7fc1faffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.645+0000 7fc1faffd640 10 monclient(hunting): dump: 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:54.647 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:54.648 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 887339605 0 0) 0x7fc1e40051d0 con 0x7fc2040a98f0 2024-03-25T18:20:54.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 >> v1:172.21.15.22:6790/0 conn(0x7fc20413f710 legacy=0x7fc204133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:54.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 >> v1:172.21.15.33:6789/0 conn(0x7fc2041430c0 legacy=0x7fc204133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:54.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:54.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 1 monclient: found mon.a 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.651+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2040abc30 con 0x7fc2040a98f0 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc1faffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc1faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc1faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 5 monclient: authenticate success, global_id 4293 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 10 monclient: _renew_subs 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2040abf10 con 0x7fc2040a98f0 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 10 monclient: _renew_subs 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:54.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc204139370 con 0x7fc2040a98f0 2024-03-25T18:20:54.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.652+0000 7fc208a1e640 1 librados: init done 2024-03-25T18:20:54.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.653+0000 7fc208a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:54.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.653+0000 7fc208a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:54.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.653+0000 7fc208a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:54.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.653+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2040627b0 con 0x7fc2040a98f0 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.961+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1f4003610 con 0x7fc2040a98f0 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.961+0000 7fc1faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.961+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc1f4003920 con 0x7fc2040a98f0 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.961+0000 7fc1faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.961+0000 7fc1faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.961+0000 7fc1faffd640 10 monclient: dump: 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:20:54.963 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:54.964 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:20:54.964 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:20:54.964 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:20:54.964 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:20:54.964 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:20:54.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.961+0000 7fc20159a640 4 set_mon_vals no callback set 2024-03-25T18:20:54.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.962+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fc1f40198f0 con 0x7fc2040a98f0 2024-03-25T18:20:54.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.963+0000 7fc1fbfff640 10 monclient: get_auth_request con 0x7fc1e406e260 auth_method 0 2024-03-25T18:20:54.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.963+0000 7fc1faffd640 10 client.4293.objecter ms_handle_connect 0x7fc1e406e260 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fc1f4053560 con 0x7fc2040a98f0 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 10 client.4293.objecter ms_dispatch 0x7fc204000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1f404f070 con 0x7fc2040a98f0 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:54.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:54.968+0000 7fc1faffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:55.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.183+0000 7fc208a1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:55.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.183+0000 7fc208a1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:55.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.183+0000 7fc208a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:20:55.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.183+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc204138980 con 0x7fc2040a98f0 2024-03-25T18:20:55.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.218+0000 7fc1faffd640 1 -- 172.21.15.22:0/3285301964 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fc1f4050fa0 con 0x7fc2040a98f0 2024-03-25T18:20:55.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.218+0000 7fc1faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:55.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.218+0000 7fc1faffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:55.220 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:20:55.220 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:55.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 10 librados: watch_flush enter 2024-03-25T18:20:55.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 10 librados: watch_flush exit 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 10 client.4293.objecter successfully canceled tick 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 >> v1:172.21.15.33:6800/215350958 conn(0x7fc1e406e260 legacy=0x7fc1e4070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 10 monclient: shutdown 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 >> v1:172.21.15.22:6789/0 conn(0x7fc2040a98f0 legacy=0x7fc204138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 shutdown_connections 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 >> 172.21.15.22:0/3285301964 conn(0x7fc2040b7ea0 msgr2=0x7fc204074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 shutdown_connections 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 1 -- 172.21.15.22:0/3285301964 wait complete. 2024-03-25T18:20:55.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:20:55.221+0000 7fc208a1e640 1 librados: shutdown 2024-03-25T18:20:55.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:02.038 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:02.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.149+0000 7fb43f0b3640 10 monclient: get_monmap_and_config 2024-03-25T18:21:02.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.149+0000 7fb43f0b3640 10 monclient: build_initial_monmap 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.149+0000 7fb43f0b3640 10 monclient: monmap: 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:02.151132+0000 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:02.151132+0000 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:02.151 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.150+0000 7fb43f0b3640 1 Processor -- start 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.150+0000 7fb43f0b3640 1 -- start start 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.150+0000 7fb43f0b3640 10 monclient: init 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb4380b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb4380b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb43814be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380af940 con 0x7fb43814be90 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380b0a70 con 0x7fb4380b4d60 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380b0d50 con 0x7fb4380b1730 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:02.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.151+0000 7fb43f0b3640 10 monclient(hunting): authenticate will time out at 7261.374974s 2024-03-25T18:21:02.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.157+0000 7fb43ce4f640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fb4380b4d60 0x7fb43813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:57710/0 (socket says 172.21.15.22:57710) 2024-03-25T18:21:02.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.157+0000 7fb43ce4f640 1 -- 172.21.15.22:0/3442359930 learned_addr learned my addr 172.21.15.22:0/3442359930 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:02.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.168+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337134987 0 0) 0x7fb4380b0d50 con 0x7fb4380b1730 2024-03-25T18:21:02.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.169+0000 7fb4377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.169+0000 7fb4377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.169+0000 7fb4377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.169+0000 7fb4377fe640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:21:02.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.169+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb428003520 con 0x7fb4380b1730 2024-03-25T18:21:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.170+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628357902 0 0) 0x7fb4380b0a70 con 0x7fb4380b4d60 2024-03-25T18:21:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.170+0000 7fb4377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.170+0000 7fb4377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.170+0000 7fb4377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.170+0000 7fb4377fe640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:21:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.170+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4380b0d50 con 0x7fb4380b4d60 2024-03-25T18:21:02.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4263987692 0 0) 0x7fb428003520 con 0x7fb4380b1730 2024-03-25T18:21:02.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 >> v1:172.21.15.33:6789/0 conn(0x7fb4380b4d60 legacy=0x7fb43813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 >> v1:172.21.15.22:6789/0 conn(0x7fb43814be90 legacy=0x7fb43814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:02.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 1 monclient: found mon.noname-b 2024-03-25T18:21:02.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:02.178 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4380b1030 con 0x7fb4380b1730 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb424001690 con 0x7fb4380b1730 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb43f0b3640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb43f0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb4377fe640 10 monclient: dump: 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.179 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.176+0000 7fb43f0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4240017f0 con 0x7fb4380b1730 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb4377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb4377fe640 1 -- 172.21.15.22:0/3442359930 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb424001c00 con 0x7fb4380b1730 2024-03-25T18:21:02.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb4377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb4377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb4377fe640 10 monclient: dump: 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb43f0b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.179+0000 7fb43f0b3640 4 set_mon_vals no callback set 2024-03-25T18:21:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.180+0000 7fb43f0b3640 10 monclient: shutdown 2024-03-25T18:21:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.180+0000 7fb43f0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.180+0000 7fb43f0b3640 1 -- 172.21.15.22:0/3442359930 >> v1:172.21.15.22:6790/0 conn(0x7fb4380b1730 legacy=0x7fb4380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.180+0000 7fb43f0b3640 1 -- 172.21.15.22:0/3442359930 shutdown_connections 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.180+0000 7fb43f0b3640 1 -- 172.21.15.22:0/3442359930 >> 172.21.15.22:0/3442359930 conn(0x7fb4380617a0 msgr2=0x7fb438061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.180+0000 7fb43f0b3640 1 -- 172.21.15.22:0/3442359930 shutdown_connections 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.180+0000 7fb43f0b3640 1 -- 172.21.15.22:0/3442359930 wait complete. 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 10 monclient: build_initial_monmap 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 10 monclient: monmap: 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:02.183 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 1 librados: starting msgr at 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 1 librados: starting objecter 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 1 Processor -- start 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 1 -- start start 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 1 librados: setting wanted keys 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 1 librados: calling monclient init 2024-03-25T18:21:02.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.181+0000 7fb43f0b3640 10 monclient: init 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fb4380b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fb4380b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fb43814be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb438137d90 con 0x7fb4380b4d60 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb438142110 con 0x7fb43814be90 2024-03-25T18:21:02.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb438140b20 con 0x7fb4380b1730 2024-03-25T18:21:02.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:02.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.182+0000 7fb43f0b3640 10 monclient(hunting): authenticate will time out at 7261.406165s 2024-03-25T18:21:02.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.194+0000 7fb43d650640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fb43814be90 0x7fb4381e3540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:57732/0 (socket says 172.21.15.22:57732) 2024-03-25T18:21:02.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.194+0000 7fb43d650640 1 -- 172.21.15.22:0/1470687420 learned_addr learned my addr 172.21.15.22:0/1470687420 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:02.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.196+0000 7fb4357fa640 10 client.?.objecter ms_handle_connect 0x7fb43814be90 2024-03-25T18:21:02.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.196+0000 7fb4357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:02.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.213+0000 7fb4357fa640 10 client.?.objecter ms_handle_connect 0x7fb4380b4d60 2024-03-25T18:21:02.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.213+0000 7fb4357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:02.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.216+0000 7fb4357fa640 10 client.?.objecter ms_handle_connect 0x7fb4380b1730 2024-03-25T18:21:02.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.216+0000 7fb4357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:02.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.222+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132964206 0 0) 0x7fb438137d90 con 0x7fb4380b4d60 2024-03-25T18:21:02.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.222+0000 7fb4357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.222+0000 7fb4357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.222+0000 7fb4357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.222+0000 7fb4357fa640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:21:02.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.222+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb41c004eb0 con 0x7fb4380b4d60 2024-03-25T18:21:02.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.223+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463463008 0 0) 0x7fb438142110 con 0x7fb43814be90 2024-03-25T18:21:02.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.223+0000 7fb4357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.223+0000 7fb4357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.223+0000 7fb4357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.223+0000 7fb4357fa640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:21:02.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.223+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb438137d90 con 0x7fb43814be90 2024-03-25T18:21:02.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.225+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211402898 0 0) 0x7fb438140b20 con 0x7fb4380b1730 2024-03-25T18:21:02.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.225+0000 7fb4357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.225+0000 7fb4357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.225+0000 7fb4357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.225+0000 7fb4357fa640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:21:02.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.225+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb438142110 con 0x7fb4380b1730 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.239+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970009742 0 0) 0x7fb438137d90 con 0x7fb43814be90 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.239+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb438140b20 con 0x7fb43814be90 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.239+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb430001c10 con 0x7fb43814be90 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.239+0000 7fb4357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.239+0000 7fb4357fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.239+0000 7fb4357fa640 10 monclient(hunting): dump: 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.240 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.241 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:02.241 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:02.241 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:02.241 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:02.241 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.239+0000 7fb4357fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:02.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.240+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 84806491 0 0) 0x7fb41c004eb0 con 0x7fb4380b4d60 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.240+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb438137d90 con 0x7fb4380b4d60 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.240+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb424001b00 con 0x7fb4380b4d60 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.240+0000 7fb4357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.240+0000 7fb4357fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.240+0000 7fb4357fa640 10 monclient(hunting): dump: 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:02.242 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.247+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3377119856 0 0) 0x7fb438142110 con 0x7fb4380b1730 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.247+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb41c004eb0 con 0x7fb4380b1730 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.248+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb42c001930 con 0x7fb4380b1730 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.248+0000 7fb4357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.248+0000 7fb4357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.248+0000 7fb4357fa640 10 monclient(hunting): dump: 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:02.249 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.250 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.250 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.250 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:02.250 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:02.250 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:02.250 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:02.250 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546628959 0 0) 0x7fb438140b20 con 0x7fb43814be90 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 >> v1:172.21.15.22:6790/0 conn(0x7fb4380b1730 legacy=0x7fb438140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 >> v1:172.21.15.22:6789/0 conn(0x7fb4380b4d60 legacy=0x7fb438133fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 1 monclient: found mon.b 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:02.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb438140cb0 con 0x7fb43814be90 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb4357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 10 monclient: _renew_subs 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb438140ef0 con 0x7fb43814be90 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 10 monclient: _renew_subs 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb438142710 con 0x7fb43814be90 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.249+0000 7fb43f0b3640 1 librados: init done 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.250+0000 7fb43f0b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.250+0000 7fb43f0b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.250+0000 7fb43f0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:02.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.250+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4380b2270 con 0x7fb43814be90 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.539+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb430003990 con 0x7fb43814be90 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.539+0000 7fb4357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.539+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb430002100 con 0x7fb43814be90 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.539+0000 7fb4357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.539+0000 7fb4357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.539+0000 7fb4357fa640 10 monclient: dump: 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:02.541 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:02.542 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:02.542 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:02.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.539+0000 7fb436ffd640 4 set_mon_vals no callback set 2024-03-25T18:21:02.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.540+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fb430003e20 con 0x7fb43814be90 2024-03-25T18:21:02.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.541+0000 7fb43ce4f640 10 monclient: get_auth_request con 0x7fb41c06e0b0 auth_method 0 2024-03-25T18:21:02.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.541+0000 7fb4357fa640 10 client.4300.objecter ms_handle_connect 0x7fb41c06e0b0 2024-03-25T18:21:02.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.547+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fb4300549c0 con 0x7fb43814be90 2024-03-25T18:21:02.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.547+0000 7fb4357fa640 10 client.4300.objecter ms_dispatch 0x7fb438000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:02.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.547+0000 7fb4357fa640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:02.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.547+0000 7fb4357fa640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:02.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.547+0000 7fb4357fa640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T18:21:02.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.547+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4300504d0 con 0x7fb43814be90 2024-03-25T18:21:02.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.548+0000 7fb4357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:02.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.548+0000 7fb4357fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:02.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.791+0000 7fb43f0b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:02.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.792+0000 7fb43f0b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:02.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.792+0000 7fb43f0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:02.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.792+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb438137f40 con 0x7fb43814be90 2024-03-25T18:21:02.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.829+0000 7fb4357fa640 1 -- 172.21.15.22:0/1470687420 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fb438137f40 con 0x7fb43814be90 2024-03-25T18:21:02.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.829+0000 7fb4357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:02.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.829+0000 7fb4357fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:02.831 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:02.831 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.831+0000 7fb43f0b3640 10 librados: watch_flush enter 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.831+0000 7fb43f0b3640 10 librados: watch_flush exit 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.831+0000 7fb43f0b3640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.831+0000 7fb43f0b3640 10 client.4300.objecter successfully canceled tick 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.831+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 >> v1:172.21.15.33:6800/215350958 conn(0x7fb41c06e0b0 legacy=0x7fb41c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.831+0000 7fb43f0b3640 10 monclient: shutdown 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.831+0000 7fb43f0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.832+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 >> v1:172.21.15.33:6789/0 conn(0x7fb43814be90 legacy=0x7fb4381e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.832+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 shutdown_connections 2024-03-25T18:21:02.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.832+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 >> 172.21.15.22:0/1470687420 conn(0x7fb4380617a0 msgr2=0x7fb43813b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:02.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.832+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 shutdown_connections 2024-03-25T18:21:02.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.832+0000 7fb43f0b3640 1 -- 172.21.15.22:0/1470687420 wait complete. 2024-03-25T18:21:02.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:02.832+0000 7fb43f0b3640 1 librados: shutdown 2024-03-25T18:21:02.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:09.652 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:09.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.766+0000 7f032da4e640 10 monclient: get_monmap_and_config 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.766+0000 7f032da4e640 10 monclient: build_initial_monmap 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.766+0000 7f032da4e640 10 monclient: monmap: 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:09.767814+0000 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:09.767814+0000 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:09.768 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:09.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.767+0000 7f032da4e640 1 Processor -- start 2024-03-25T18:21:09.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.767+0000 7f032da4e640 1 -- start start 2024-03-25T18:21:09.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.767+0000 7f032da4e640 10 monclient: init 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.767+0000 7f032da4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.767+0000 7f032da4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.767+0000 7f032da4e640 10 monclient(hunting): picked mon.noname-c con 0x7f03280a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.767+0000 7f032da4e640 10 monclient(hunting): picked mon.noname-b con 0x7f032813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.768+0000 7f032da4e640 10 monclient(hunting): picked mon.noname-a con 0x7f03281430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.768+0000 7f032da4e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0328142520 con 0x7f03281430c0 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.768+0000 7f032da4e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280b28a0 con 0x7f03280a98f0 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.768+0000 7f032da4e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03280b2a30 con 0x7f032813f710 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.768+0000 7f032da4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:09.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.768+0000 7f032da4e640 10 monclient(hunting): authenticate will time out at 7268.991745s 2024-03-25T18:21:09.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.776+0000 7f0326ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f03280a98f0 0x7f03280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:34050/0 (socket says 172.21.15.22:34050) 2024-03-25T18:21:09.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.776+0000 7f0326ffd640 1 -- 172.21.15.22:0/1679963573 learned_addr learned my addr 172.21.15.22:0/1679963573 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:09.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.799+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181491375 0 0) 0x7f0328142520 con 0x7f03281430c0 2024-03-25T18:21:09.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.799+0000 7f0325ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.799+0000 7f0325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.799+0000 7f0325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.799+0000 7f0325ffb640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:21:09.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.799+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310003520 con 0x7f03281430c0 2024-03-25T18:21:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.801+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352826268 0 0) 0x7f03280b28a0 con 0x7f03280a98f0 2024-03-25T18:21:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.801+0000 7f0325ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.801+0000 7f0325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.801+0000 7f0325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.801+0000 7f0325ffb640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:21:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.801+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0328142520 con 0x7f03280a98f0 2024-03-25T18:21:09.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.804+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367944375 0 0) 0x7f03280b2a30 con 0x7f032813f710 2024-03-25T18:21:09.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.804+0000 7f0325ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.804+0000 7f0325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.804+0000 7f0325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.804+0000 7f0325ffb640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:21:09.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.804+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03280b28a0 con 0x7f032813f710 2024-03-25T18:21:09.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4041346990 0 0) 0x7f0328142520 con 0x7f03280a98f0 2024-03-25T18:21:09.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 >> v1:172.21.15.22:6790/0 conn(0x7f032813f710 legacy=0x7f03281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 >> v1:172.21.15.22:6789/0 conn(0x7f03281430c0 legacy=0x7f032814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:09.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 1 monclient: found mon.noname-c 2024-03-25T18:21:09.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280b44c0 con 0x7f03280a98f0 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0318001670 con 0x7f03280a98f0 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f0325ffb640 10 monclient: dump: 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:09.811 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:09.812 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:09.812 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:09.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f032da4e640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:21:09.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.809+0000 7f032da4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03180017d0 con 0x7f03280a98f0 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f0325ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f0325ffb640 1 -- 172.21.15.22:0/1679963573 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0318001be0 con 0x7f03280a98f0 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f0325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f0325ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f0325ffb640 10 monclient: dump: 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.824 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.825 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:09.825 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:09.825 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:09.825 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:09.825 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f032da4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f032da4e640 4 set_mon_vals no callback set 2024-03-25T18:21:09.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f032da4e640 10 monclient: shutdown 2024-03-25T18:21:09.826 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f032da4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.823+0000 7f032da4e640 1 -- 172.21.15.22:0/1679963573 >> v1:172.21.15.33:6789/0 conn(0x7f03280a98f0 legacy=0x7f03280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.824+0000 7f032da4e640 1 -- 172.21.15.22:0/1679963573 shutdown_connections 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.824+0000 7f032da4e640 1 -- 172.21.15.22:0/1679963573 >> 172.21.15.22:0/1679963573 conn(0x7f03280b7ea0 msgr2=0x7f03280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.824+0000 7f032da4e640 1 -- 172.21.15.22:0/1679963573 shutdown_connections 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.824+0000 7f032da4e640 1 -- 172.21.15.22:0/1679963573 wait complete. 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.824+0000 7f032da4e640 10 monclient: build_initial_monmap 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.824+0000 7f032da4e640 10 monclient: monmap: 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:09.827 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.825+0000 7f032da4e640 1 librados: starting msgr at 2024-03-25T18:21:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.825+0000 7f032da4e640 1 librados: starting objecter 2024-03-25T18:21:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.825+0000 7f032da4e640 1 Processor -- start 2024-03-25T18:21:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.825+0000 7f032da4e640 1 -- start start 2024-03-25T18:21:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.825+0000 7f032da4e640 1 librados: setting wanted keys 2024-03-25T18:21:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.825+0000 7f032da4e640 1 librados: calling monclient init 2024-03-25T18:21:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.825+0000 7f032da4e640 10 monclient: init 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 10 monclient(hunting): picked mon.noname-a con 0x7f03280a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 10 monclient(hunting): picked mon.noname-c con 0x7f032813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 10 monclient(hunting): picked mon.noname-b con 0x7f03281430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03281390f0 con 0x7f03280a98f0 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0328138dd0 con 0x7f03281430c0 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03281346b0 con 0x7f032813f710 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.826+0000 7f032da4e640 10 monclient(hunting): authenticate will time out at 7269.050371s 2024-03-25T18:21:09.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.836+0000 7f03267fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f032813f710 0x7f03281386c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41624/0 (socket says 172.21.15.22:41624) 2024-03-25T18:21:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.837+0000 7f03267fc640 1 -- 172.21.15.22:0/636795482 learned_addr learned my addr 172.21.15.22:0/636795482 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.837+0000 7f030ffff640 10 client.?.objecter ms_handle_connect 0x7f032813f710 2024-03-25T18:21:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.837+0000 7f030ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:09.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 10 client.?.objecter ms_handle_connect 0x7f03281430c0 2024-03-25T18:21:09.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393261079 0 0) 0x7f03281346b0 con 0x7f032813f710 2024-03-25T18:21:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:21:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.845+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0314004eb0 con 0x7f032813f710 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 10 client.?.objecter ms_handle_connect 0x7f03280a98f0 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273462576 0 0) 0x7f0328138dd0 con 0x7f03281430c0 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:21:09.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.856+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03281346b0 con 0x7f03281430c0 2024-03-25T18:21:09.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.859+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1467397252 0 0) 0x7f0314004eb0 con 0x7f032813f710 2024-03-25T18:21:09.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.859+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0328138dd0 con 0x7f032813f710 2024-03-25T18:21:09.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.859+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f031c001a10 con 0x7f032813f710 2024-03-25T18:21:09.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.859+0000 7f030ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.859+0000 7f030ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:09.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.859+0000 7f030ffff640 10 monclient(hunting): dump: 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:09.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.859+0000 7f030ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:09.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.867+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1842788813 0 0) 0x7f03281346b0 con 0x7f03281430c0 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.867+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0314004eb0 con 0x7f03281430c0 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.867+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0320001a70 con 0x7f03281430c0 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.867+0000 7f030ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.867+0000 7f030ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.867+0000 7f030ffff640 10 monclient(hunting): dump: 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:09.869 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:09.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.868+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037227300 0 0) 0x7f03281390f0 con 0x7f03280a98f0 2024-03-25T18:21:09.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.868+0000 7f030ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:09.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.868+0000 7f030ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:09.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.868+0000 7f030ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:09.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.868+0000 7f030ffff640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:21:09.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.868+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03281346b0 con 0x7f03280a98f0 2024-03-25T18:21:09.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1545706260 0 0) 0x7f0314004eb0 con 0x7f03281430c0 2024-03-25T18:21:09.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 >> v1:172.21.15.22:6790/0 conn(0x7f032813f710 legacy=0x7f03281386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 >> v1:172.21.15.22:6789/0 conn(0x7f03280a98f0 legacy=0x7f0328137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 1 monclient: found mon.b 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0328134840 con 0x7f03281430c0 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 10 monclient: _finish_auth 0 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f030ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f032da4e640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f032da4e640 10 monclient: _renew_subs 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f032da4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.873+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0328134a90 con 0x7f03281430c0 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.874+0000 7f032da4e640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.874+0000 7f032da4e640 10 monclient: _renew_subs 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.874+0000 7f032da4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:09.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.874+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03281356f0 con 0x7f03281430c0 2024-03-25T18:21:09.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.874+0000 7f032da4e640 1 librados: init done 2024-03-25T18:21:09.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.875+0000 7f032da4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:09.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.875+0000 7f032da4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:09.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.875+0000 7f032da4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:09.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:09.875+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03280627b0 con 0x7f03281430c0 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.158+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0320001f60 con 0x7f03281430c0 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.158+0000 7f030ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.158+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0320003b30 con 0x7f03281430c0 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.158+0000 7f030ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.158+0000 7f030ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.158+0000 7f030ffff640 10 monclient: dump: 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:10.159 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.158+0000 7f03257fa640 4 set_mon_vals no callback set 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.159+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f0320003d50 con 0x7f03281430c0 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.159+0000 7f0326ffd640 10 monclient: get_auth_request con 0x7f031406df30 auth_method 0 2024-03-25T18:21:10.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.160+0000 7f030ffff640 10 client.4306.objecter ms_handle_connect 0x7f031406df30 2024-03-25T18:21:10.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.165+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f0320053700 con 0x7f03281430c0 2024-03-25T18:21:10.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.165+0000 7f030ffff640 10 client.4306.objecter ms_dispatch 0x7f0328000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:10.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.165+0000 7f030ffff640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:10.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.165+0000 7f030ffff640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:10.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.165+0000 7f030ffff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:21:10.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.166+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f032004f210 con 0x7f03281430c0 2024-03-25T18:21:10.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.166+0000 7f030ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:10.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.166+0000 7f030ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:10.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.392+0000 7f032da4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:10.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.392+0000 7f032da4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:10.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.392+0000 7f032da4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:10.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.392+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03281361b0 con 0x7f03281430c0 2024-03-25T18:21:10.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.420+0000 7f030ffff640 1 -- 172.21.15.22:0/636795482 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f0320051140 con 0x7f03281430c0 2024-03-25T18:21:10.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.420+0000 7f030ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:10.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.420+0000 7f030ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:10.422 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:10.422 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 10 librados: watch_flush enter 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 10 librados: watch_flush exit 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 >> v1:172.21.15.33:6800/215350958 conn(0x7f031406df30 legacy=0x7f03140703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 10 monclient: shutdown 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 >> v1:172.21.15.33:6789/0 conn(0x7f03281430c0 legacy=0x7f0328133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 shutdown_connections 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.423+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 >> 172.21.15.22:0/636795482 conn(0x7f03280b7ea0 msgr2=0x7f0328074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:10.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.424+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 shutdown_connections 2024-03-25T18:21:10.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.424+0000 7f032da4e640 1 -- 172.21.15.22:0/636795482 wait complete. 2024-03-25T18:21:10.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:10.424+0000 7f032da4e640 1 librados: shutdown 2024-03-25T18:21:10.436 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:12.171 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:14.833 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.224 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:14.886 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.257 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:14.919 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.298 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:14.916 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.299 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:14.961 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.306 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:14.969 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.312 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:14.930 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.333 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:14.951 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:12.345 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:15.007 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:17.243 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:17.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.358+0000 7f35f6bcf640 10 monclient: get_monmap_and_config 2024-03-25T18:21:17.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.358+0000 7f35f6bcf640 10 monclient: build_initial_monmap 2024-03-25T18:21:17.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.358+0000 7f35f6bcf640 10 monclient: monmap: 2024-03-25T18:21:17.359 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:17.359 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:17.359833+0000 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:17.359833+0000 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:17.360 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.359+0000 7f35f6bcf640 1 Processor -- start 2024-03-25T18:21:17.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.359+0000 7f35f6bcf640 1 -- start start 2024-03-25T18:21:17.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.359+0000 7f35f6bcf640 10 monclient: init 2024-03-25T18:21:17.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.359+0000 7f35f6bcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:17.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.359+0000 7f35f6bcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:17.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.359+0000 7f35f6bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f35f00a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:17.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.359+0000 7f35f6bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f35f013f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:17.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.360+0000 7f35f6bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f35f01430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:17.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.360+0000 7f35f6bcf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f0142520 con 0x7f35f01430c0 2024-03-25T18:21:17.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.360+0000 7f35f6bcf640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00b28a0 con 0x7f35f013f710 2024-03-25T18:21:17.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.360+0000 7f35f6bcf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00b2a30 con 0x7f35f00a98f0 2024-03-25T18:21:17.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.360+0000 7f35f6bcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:17.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.360+0000 7f35f6bcf640 10 monclient(hunting): authenticate will time out at 7276.583652s 2024-03-25T18:21:17.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.367+0000 7f35f496b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f35f00a98f0 0x7f35f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37578/0 (socket says 172.21.15.22:37578) 2024-03-25T18:21:17.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.367+0000 7f35f496b640 1 -- 172.21.15.22:0/1906937918 learned_addr learned my addr 172.21.15.22:0/1906937918 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.375+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174605213 0 0) 0x7f35f00b2a30 con 0x7f35f00a98f0 2024-03-25T18:21:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.375+0000 7f35ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.375+0000 7f35ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.375+0000 7f35ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.375+0000 7f35ef7fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:21:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.375+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e0003520 con 0x7f35f00a98f0 2024-03-25T18:21:17.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677726964 0 0) 0x7f35e0003520 con 0x7f35f00a98f0 2024-03-25T18:21:17.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 >> v1:172.21.15.33:6789/0 conn(0x7f35f013f710 legacy=0x7f35f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:17.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 >> v1:172.21.15.22:6789/0 conn(0x7f35f01430c0 legacy=0x7f35f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:17.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:17.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 1 monclient: found mon.noname-b 2024-03-25T18:21:17.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00b44c0 con 0x7f35f00a98f0 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f35e4001690 con 0x7f35f00a98f0 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35ef7fe640 10 monclient: dump: 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:17.392 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:17.393 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35f6bcf640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:21:17.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.389+0000 7f35f6bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e40017f0 con 0x7f35f00a98f0 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35ef7fe640 1 -- 172.21.15.22:0/1906937918 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f35e4001c00 con 0x7f35f00a98f0 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35f6bcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35f6bcf640 4 set_mon_vals no callback set 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.423+0000 7f35ef7fe640 10 monclient: dump: 2024-03-25T18:21:17.424 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:17.425 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.424+0000 7f35f6bcf640 10 monclient: shutdown 2024-03-25T18:21:17.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.424+0000 7f35f6bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:17.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.424+0000 7f35f6bcf640 1 -- 172.21.15.22:0/1906937918 >> v1:172.21.15.22:6790/0 conn(0x7f35f00a98f0 legacy=0x7f35f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.424+0000 7f35f6bcf640 1 -- 172.21.15.22:0/1906937918 shutdown_connections 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.424+0000 7f35f6bcf640 1 -- 172.21.15.22:0/1906937918 >> 172.21.15.22:0/1906937918 conn(0x7f35f00b7ea0 msgr2=0x7f35f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.424+0000 7f35f6bcf640 1 -- 172.21.15.22:0/1906937918 shutdown_connections 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.424+0000 7f35f6bcf640 1 -- 172.21.15.22:0/1906937918 wait complete. 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 10 monclient: build_initial_monmap 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 10 monclient: monmap: 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 1 librados: starting msgr at 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 1 librados: starting objecter 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 1 Processor -- start 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 1 -- start start 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 1 librados: setting wanted keys 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 1 librados: calling monclient init 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.425+0000 7f35f6bcf640 10 monclient: init 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:17.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 10 monclient(hunting): picked mon.noname-b con 0x7f35f00a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 10 monclient(hunting): picked mon.noname-c con 0x7f35f013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 10 monclient(hunting): picked mon.noname-a con 0x7f35f01430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f0139a40 con 0x7f35f01430c0 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f0133f20 con 0x7f35f00a98f0 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00ab950 con 0x7f35f013f710 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:17.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.426+0000 7f35f6bcf640 10 monclient(hunting): authenticate will time out at 7276.650535s 2024-03-25T18:21:17.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.434+0000 7f35f516c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f35f01430c0 0x7f35f0133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48884/0 (socket says 172.21.15.22:48884) 2024-03-25T18:21:17.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.434+0000 7f35f516c640 1 -- 172.21.15.22:0/2964091438 learned_addr learned my addr 172.21.15.22:0/2964091438 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:17.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.436+0000 7f35ed7fa640 10 client.?.objecter ms_handle_connect 0x7f35f01430c0 2024-03-25T18:21:17.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.436+0000 7f35ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:17.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.437+0000 7f35ed7fa640 10 client.?.objecter ms_handle_connect 0x7f35f00a98f0 2024-03-25T18:21:17.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.437+0000 7f35ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.440+0000 7f35ed7fa640 10 client.?.objecter ms_handle_connect 0x7f35f013f710 2024-03-25T18:21:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.440+0000 7f35ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:17.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.442+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650012100 0 0) 0x7f35f0133f20 con 0x7f35f00a98f0 2024-03-25T18:21:17.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.442+0000 7f35ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:17.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.442+0000 7f35ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:17.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.442+0000 7f35ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:17.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.442+0000 7f35ed7fa640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:21:17.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.442+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d4004eb0 con 0x7f35f00a98f0 2024-03-25T18:21:17.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.449+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815867486 0 0) 0x7f35f0139a40 con 0x7f35f01430c0 2024-03-25T18:21:17.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.449+0000 7f35ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:17.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.449+0000 7f35ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:17.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.449+0000 7f35ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:17.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.449+0000 7f35ed7fa640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:21:17.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.449+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f0133f20 con 0x7f35f01430c0 2024-03-25T18:21:17.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.453+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851840611 0 0) 0x7f35f00ab950 con 0x7f35f013f710 2024-03-25T18:21:17.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.453+0000 7f35ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:17.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.453+0000 7f35ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:17.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.453+0000 7f35ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:17.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.453+0000 7f35ed7fa640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:21:17.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.453+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f0139a40 con 0x7f35f013f710 2024-03-25T18:21:17.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2332715971 0 0) 0x7f35d4004eb0 con 0x7f35f00a98f0 2024-03-25T18:21:17.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35f00ab950 con 0x7f35f00a98f0 2024-03-25T18:21:17.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f35e4001bc0 con 0x7f35f00a98f0 2024-03-25T18:21:17.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:17.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:17.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:17.467 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 7904505 0 0) 0x7f35f0133f20 con 0x7f35f01430c0 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35d4004eb0 con 0x7f35f01430c0 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f35e8001c10 con 0x7f35f01430c0 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.466+0000 7f35ed7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:17.468 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:17.469 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.475+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 236396167 0 0) 0x7f35f0139a40 con 0x7f35f013f710 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.475+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35f0133f20 con 0x7f35f013f710 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.475+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f35dc001860 con 0x7f35f013f710 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.475+0000 7f35ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.475+0000 7f35ed7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.475+0000 7f35ed7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:17.477 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:17.478 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:17.478 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:17.478 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.478+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3922780173 0 0) 0x7f35f00ab950 con 0x7f35f00a98f0 2024-03-25T18:21:17.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 >> v1:172.21.15.22:6790/0 conn(0x7f35f013f710 legacy=0x7f35f0133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 >> v1:172.21.15.22:6789/0 conn(0x7f35f01430c0 legacy=0x7f35f0133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 1 monclient: found mon.b 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f00abc30 con 0x7f35f00a98f0 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 10 monclient: _renew_subs 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f00abf10 con 0x7f35f00a98f0 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 10 monclient: _renew_subs 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:17.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f0139370 con 0x7f35f00a98f0 2024-03-25T18:21:17.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.479+0000 7f35f6bcf640 1 librados: init done 2024-03-25T18:21:17.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.480+0000 7f35f6bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:17.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.480+0000 7f35f6bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:17.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.480+0000 7f35f6bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:17.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.480+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f00627b0 con 0x7f35f00a98f0 2024-03-25T18:21:17.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.802+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35e4004a90 con 0x7f35f00a98f0 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.802+0000 7f35ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.802+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f35e4004da0 con 0x7f35f00a98f0 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.802+0000 7f35ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.802+0000 7f35ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.802+0000 7f35ed7fa640 10 monclient: dump: 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:17.804 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:17.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.802+0000 7f35eeffd640 4 set_mon_vals no callback set 2024-03-25T18:21:17.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.803+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f35e401ad70 con 0x7f35f00a98f0 2024-03-25T18:21:17.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.804+0000 7f35effff640 10 monclient: get_auth_request con 0x7f35d406e0f0 auth_method 0 2024-03-25T18:21:17.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.804+0000 7f35ed7fa640 10 client.4312.objecter ms_handle_connect 0x7f35d406e0f0 2024-03-25T18:21:17.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f35e4054930 con 0x7f35f00a98f0 2024-03-25T18:21:17.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 10 client.4312.objecter ms_dispatch 0x7f35f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:17.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:17.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:17.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T18:21:17.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35e4050270 con 0x7f35f00a98f0 2024-03-25T18:21:17.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:17.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:17.808+0000 7f35ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:18.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.041+0000 7f35f6bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:18.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.041+0000 7f35f6bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:18.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.041+0000 7f35f6bcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:18.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.041+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35f00b7180 con 0x7f35f00a98f0 2024-03-25T18:21:18.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.070+0000 7f35ed7fa640 1 -- 172.21.15.22:0/2964091438 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f35e4052370 con 0x7f35f00a98f0 2024-03-25T18:21:18.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.070+0000 7f35ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:18.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.070+0000 7f35ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:18.071 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:18.071 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 10 librados: watch_flush enter 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 10 librados: watch_flush exit 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 10 client.4312.objecter successfully canceled tick 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 >> v1:172.21.15.33:6800/215350958 conn(0x7f35d406e0f0 legacy=0x7f35d40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 10 monclient: shutdown 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 >> v1:172.21.15.33:6789/0 conn(0x7f35f00a98f0 legacy=0x7f35f0138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 shutdown_connections 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 >> 172.21.15.22:0/2964091438 conn(0x7f35f00b7ea0 msgr2=0x7f35f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:18.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 shutdown_connections 2024-03-25T18:21:18.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 1 -- 172.21.15.22:0/2964091438 wait complete. 2024-03-25T18:21:18.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:18.072+0000 7f35f6bcf640 1 librados: shutdown 2024-03-25T18:21:18.085 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:24.892 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:25.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.007+0000 7f0908d10640 10 monclient: get_monmap_and_config 2024-03-25T18:21:25.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.007+0000 7f0908d10640 10 monclient: build_initial_monmap 2024-03-25T18:21:25.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.007+0000 7f0908d10640 10 monclient: monmap: 2024-03-25T18:21:25.007 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:25.007 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:25.007631+0000 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:25.007631+0000 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:25.008 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 1 Processor -- start 2024-03-25T18:21:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 1 -- start start 2024-03-25T18:21:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient: init 2024-03-25T18:21:25.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient(hunting): picked mon.noname-a con 0x7f09040a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient(hunting): picked mon.noname-b con 0x7f090413f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient(hunting): picked mon.noname-c con 0x7f09041430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0904142520 con 0x7f09040a98f0 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09040b28a0 con 0x7f09041430c0 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09040b2a30 con 0x7f090413f710 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:25.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.008+0000 7f0908d10640 10 monclient(hunting): authenticate will time out at 7284.231500s 2024-03-25T18:21:25.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.018+0000 7f0902d9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f09041430c0 0x7f090414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:40280/0 (socket says 172.21.15.22:40280) 2024-03-25T18:21:25.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.018+0000 7f0902d9d640 1 -- 172.21.15.22:0/1945202409 learned_addr learned my addr 172.21.15.22:0/1945202409 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:25.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.033+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421122449 0 0) 0x7f09040b28a0 con 0x7f09041430c0 2024-03-25T18:21:25.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.033+0000 7f0901d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.033+0000 7f0901d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.033+0000 7f0901d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.033+0000 7f0901d9b640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:21:25.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.033+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0003520 con 0x7f09041430c0 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580053015 0 0) 0x7f08f0003520 con 0x7f09041430c0 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 >> v1:172.21.15.22:6790/0 conn(0x7f090413f710 legacy=0x7f09041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 >> v1:172.21.15.22:6789/0 conn(0x7f09040a98f0 legacy=0x7f09040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 1 monclient: found mon.noname-c 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09040b44c0 con 0x7f09041430c0 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 10 monclient: _finish_auth 0 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f08fc001a10 con 0x7f09041430c0 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0901d9b640 10 monclient: dump: 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0908d10640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:21:25.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.043+0000 7f0908d10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:25.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.048+0000 7f0901d9b640 1 -- 172.21.15.22:0/1945202409 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08fc001b70 con 0x7f09041430c0 2024-03-25T18:21:25.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.048+0000 7f0901d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:25.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.049+0000 7f0908d10640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:25.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.049+0000 7f0908d10640 4 set_mon_vals no callback set 2024-03-25T18:21:25.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.049+0000 7f0908d10640 10 monclient: shutdown 2024-03-25T18:21:25.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.049+0000 7f0908d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:25.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.049+0000 7f0908d10640 1 -- 172.21.15.22:0/1945202409 >> v1:172.21.15.33:6789/0 conn(0x7f09041430c0 legacy=0x7f090414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.050+0000 7f0908d10640 1 -- 172.21.15.22:0/1945202409 shutdown_connections 2024-03-25T18:21:25.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.050+0000 7f0908d10640 1 -- 172.21.15.22:0/1945202409 >> 172.21.15.22:0/1945202409 conn(0x7f09040b7ea0 msgr2=0x7f09040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:25.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.050+0000 7f0908d10640 1 -- 172.21.15.22:0/1945202409 shutdown_connections 2024-03-25T18:21:25.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.050+0000 7f0908d10640 1 -- 172.21.15.22:0/1945202409 wait complete. 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 10 monclient: build_initial_monmap 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 10 monclient: monmap: 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:25.051 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 1 librados: starting msgr at 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 1 librados: starting objecter 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 1 Processor -- start 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 1 -- start start 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 1 librados: setting wanted keys 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 1 librados: calling monclient init 2024-03-25T18:21:25.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.051+0000 7f0908d10640 10 monclient: init 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 10 monclient(hunting): picked mon.noname-b con 0x7f09040a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 10 monclient(hunting): picked mon.noname-a con 0x7f090413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 10 monclient(hunting): picked mon.noname-c con 0x7f09041430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09040abca0 con 0x7f090413f710 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0904138bc0 con 0x7f09040a98f0 2024-03-25T18:21:25.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09041db250 con 0x7f09041430c0 2024-03-25T18:21:25.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:25.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.052+0000 7f0908d10640 10 monclient(hunting): authenticate will time out at 7284.274801s 2024-03-25T18:21:25.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.062+0000 7f090259c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f09040a98f0 0x7f09041383d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:40328/0 (socket says 172.21.15.22:40328) 2024-03-25T18:21:25.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.062+0000 7f090259c640 1 -- 172.21.15.22:0/1450284313 learned_addr learned my addr 172.21.15.22:0/1450284313 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:25.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.069+0000 7f08faffd640 10 client.?.objecter ms_handle_connect 0x7f09040a98f0 2024-03-25T18:21:25.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.069+0000 7f08faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:25.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.078+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955392248 0 0) 0x7f0904138bc0 con 0x7f09040a98f0 2024-03-25T18:21:25.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.078+0000 7f08faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.078+0000 7f08faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.078+0000 7f08faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.078+0000 7f08faffd640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:21:25.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.078+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08e4004eb0 con 0x7f09040a98f0 2024-03-25T18:21:25.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.079+0000 7f08faffd640 10 client.?.objecter ms_handle_connect 0x7f09041430c0 2024-03-25T18:21:25.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.079+0000 7f08faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:25.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.082+0000 7f08faffd640 10 client.?.objecter ms_handle_connect 0x7f090413f710 2024-03-25T18:21:25.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.082+0000 7f08faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:25.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.087+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272634395 0 0) 0x7f09041db250 con 0x7f09041430c0 2024-03-25T18:21:25.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.087+0000 7f08faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.087+0000 7f08faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.087+0000 7f08faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.087+0000 7f08faffd640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:21:25.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.087+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904138bc0 con 0x7f09041430c0 2024-03-25T18:21:25.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.094+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 16220564 0 0) 0x7f08e4004eb0 con 0x7f09040a98f0 2024-03-25T18:21:25.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.094+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09041db250 con 0x7f09040a98f0 2024-03-25T18:21:25.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.094+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f08f40019d0 con 0x7f09040a98f0 2024-03-25T18:21:25.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.094+0000 7f08faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.094+0000 7f08faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:25.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.094+0000 7f08faffd640 10 monclient(hunting): dump: 2024-03-25T18:21:25.094 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:25.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.094+0000 7f08faffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:25.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.096+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084843851 0 0) 0x7f0904138bc0 con 0x7f09041430c0 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.096+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08e4004eb0 con 0x7f09041430c0 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.096+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f08fc001e80 con 0x7f09041430c0 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.096+0000 7f08faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.096+0000 7f08faffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.096+0000 7f08faffd640 10 monclient(hunting): dump: 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:25.097 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:25.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3012330255 0 0) 0x7f08e4004eb0 con 0x7f09041430c0 2024-03-25T18:21:25.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 >> v1:172.21.15.33:6789/0 conn(0x7f09040a98f0 legacy=0x7f09041383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 >> v1:172.21.15.22:6789/0 conn(0x7f090413f710 legacy=0x7f0904137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 1 monclient: found mon.c 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09041db3e0 con 0x7f09041430c0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f08faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 5 monclient: authenticate success, global_id 4322 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 10 monclient: _renew_subs 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09041db630 con 0x7f09041430c0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 10 monclient: _renew_subs 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09041e3120 con 0x7f09041430c0 2024-03-25T18:21:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.107+0000 7f0908d10640 1 librados: init done 2024-03-25T18:21:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.108+0000 7f0908d10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.108+0000 7f0908d10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.108+0000 7f0908d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.108+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09040627b0 con 0x7f09041430c0 2024-03-25T18:21:25.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08fc0020a0 con 0x7f09041430c0 2024-03-25T18:21:25.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f08faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:25.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f08fc0015d0 con 0x7f09041430c0 2024-03-25T18:21:25.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f08faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f08faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f08faffd640 10 monclient: dump: 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:25.436 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:25.437 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:25.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f090159a640 4 set_mon_vals no callback set 2024-03-25T18:21:25.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.435+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f08fc01aee0 con 0x7f09041430c0 2024-03-25T18:21:25.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.436+0000 7f090259c640 10 monclient: get_auth_request con 0x7f08e406dec0 auth_method 0 2024-03-25T18:21:25.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.436+0000 7f08faffd640 10 client.4322.objecter ms_handle_connect 0x7f08e406dec0 2024-03-25T18:21:25.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f08fc054b20 con 0x7f09041430c0 2024-03-25T18:21:25.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 10 client.4322.objecter ms_dispatch 0x7f0904000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:25.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:25.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:25.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-25T18:21:25.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08fc050460 con 0x7f09041430c0 2024-03-25T18:21:25.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:25.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.441+0000 7f08faffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:25.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.687+0000 7f0908d10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:25.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.687+0000 7f0908d10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:25.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.687+0000 7f0908d10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:25.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.687+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0904074cc0 con 0x7f09041430c0 2024-03-25T18:21:25.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.713+0000 7f08faffd640 1 -- 172.21.15.22:0/1450284313 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f08fc052560 con 0x7f09041430c0 2024-03-25T18:21:25.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.713+0000 7f08faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:25.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.713+0000 7f08faffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:25.714 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:25.714 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 10 librados: watch_flush enter 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 10 librados: watch_flush exit 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 10 client.4322.objecter successfully canceled tick 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 >> v1:172.21.15.33:6800/215350958 conn(0x7f08e406dec0 legacy=0x7f08e4070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 10 monclient: shutdown 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 >> v1:172.21.15.22:6790/0 conn(0x7f09041430c0 legacy=0x7f09041dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 shutdown_connections 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 >> 172.21.15.22:0/1450284313 conn(0x7f09040b7ea0 msgr2=0x7f0904074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 shutdown_connections 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 1 -- 172.21.15.22:0/1450284313 wait complete. 2024-03-25T18:21:25.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:25.716+0000 7f0908d10640 1 librados: shutdown 2024-03-25T18:21:25.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:32.281 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:34.943 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.288 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:34.950 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.323 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:34.985 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.341 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:34.959 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.344 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:34.962 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.351 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:35.013 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.357 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:35.019 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.382 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:35.044 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.383 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:35.001 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.534 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.648+0000 7fe31432b640 10 monclient: get_monmap_and_config 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.648+0000 7fe31432b640 10 monclient: build_initial_monmap 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.648+0000 7fe31432b640 10 monclient: monmap: 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:32.649090+0000 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:32.649090+0000 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:32.649 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.649+0000 7fe31432b640 1 Processor -- start 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.649+0000 7fe31432b640 1 -- start start 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.649+0000 7fe31432b640 10 monclient: init 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 10 monclient(hunting): picked mon.noname-b con 0x7fe30c0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 10 monclient(hunting): picked mon.noname-c con 0x7fe30c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 10 monclient(hunting): picked mon.noname-a con 0x7fe30c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c142520 con 0x7fe30c1430c0 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c0b28a0 con 0x7fe30c13f710 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c0b2a30 con 0x7fe30c0a98f0 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:32.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.650+0000 7fe31432b640 10 monclient(hunting): authenticate will time out at 7291.873048s 2024-03-25T18:21:32.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.662+0000 7fe3118c6640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fe30c13f710 0x7fe30c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:44742/0 (socket says 172.21.15.22:44742) 2024-03-25T18:21:32.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.662+0000 7fe3118c6640 1 -- 172.21.15.22:0/1919644180 learned_addr learned my addr 172.21.15.22:0/1919644180 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:32.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.670+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159901290 0 0) 0x7fe30c0b28a0 con 0x7fe30c13f710 2024-03-25T18:21:32.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.670+0000 7fe3110c5640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.670+0000 7fe3110c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.670+0000 7fe3110c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.670+0000 7fe3110c5640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:21:32.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.670+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f8003520 con 0x7fe30c13f710 2024-03-25T18:21:32.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.678+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988533410 0 0) 0x7fe30c142520 con 0x7fe30c1430c0 2024-03-25T18:21:32.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.678+0000 7fe3110c5640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.678+0000 7fe3110c5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.678+0000 7fe3110c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.678+0000 7fe3110c5640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:21:32.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.678+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe30c0b28a0 con 0x7fe30c1430c0 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2661492006 0 0) 0x7fe2f8003520 con 0x7fe30c13f710 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 >> v1:172.21.15.22:6790/0 conn(0x7fe30c0a98f0 legacy=0x7fe30c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 >> v1:172.21.15.22:6789/0 conn(0x7fe30c1430c0 legacy=0x7fe30c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 10 monclient: _finish_hunting 0 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 1 monclient: found mon.noname-c 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:21:32.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c0b44c0 con 0x7fe30c13f710 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 10 monclient: _finish_auth 0 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 10 monclient: _check_auth_tickets 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe2fc001690 con 0x7fe30c13f710 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe31432b640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe31432b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.681+0000 7fe3110c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.682+0000 7fe3110c5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.682+0000 7fe3110c5640 10 monclient: dump: 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:32.683 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:32.684 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:32.684 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:32.684 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:32.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.682+0000 7fe31432b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:32.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2fc0017f0 con 0x7fe30c13f710 2024-03-25T18:21:32.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe3110c5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:32.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe3110c5640 1 -- 172.21.15.22:0/1919644180 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe2fc001c00 con 0x7fe30c13f710 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe3110c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe3110c5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe3110c5640 10 monclient: dump: 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe31432b640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:32.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.711+0000 7fe31432b640 4 set_mon_vals no callback set 2024-03-25T18:21:32.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.712+0000 7fe31432b640 10 monclient: shutdown 2024-03-25T18:21:32.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.712+0000 7fe31432b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:32.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.712+0000 7fe31432b640 1 -- 172.21.15.22:0/1919644180 >> v1:172.21.15.33:6789/0 conn(0x7fe30c13f710 legacy=0x7fe30c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.712+0000 7fe31432b640 1 -- 172.21.15.22:0/1919644180 shutdown_connections 2024-03-25T18:21:32.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.712+0000 7fe31432b640 1 -- 172.21.15.22:0/1919644180 >> 172.21.15.22:0/1919644180 conn(0x7fe30c0b7ea0 msgr2=0x7fe30c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:32.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.712+0000 7fe31432b640 1 -- 172.21.15.22:0/1919644180 shutdown_connections 2024-03-25T18:21:32.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.712+0000 7fe31432b640 1 -- 172.21.15.22:0/1919644180 wait complete. 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.713+0000 7fe31432b640 10 monclient: build_initial_monmap 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.713+0000 7fe31432b640 10 monclient: monmap: 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:32.714 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:32.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.713+0000 7fe31432b640 1 librados: starting msgr at 2024-03-25T18:21:32.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.713+0000 7fe31432b640 1 librados: starting objecter 2024-03-25T18:21:32.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 1 Processor -- start 2024-03-25T18:21:32.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 1 -- start start 2024-03-25T18:21:32.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 1 librados: setting wanted keys 2024-03-25T18:21:32.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 1 librados: calling monclient init 2024-03-25T18:21:32.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 10 monclient: init 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.714+0000 7fe31432b640 10 monclient(hunting): picked mon.noname-a con 0x7fe30c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.715+0000 7fe31432b640 10 monclient(hunting): picked mon.noname-c con 0x7fe30c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.715+0000 7fe31432b640 10 monclient(hunting): picked mon.noname-b con 0x7fe30c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.715+0000 7fe31432b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c06ef90 con 0x7fe30c0a98f0 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.715+0000 7fe31432b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c06f9e0 con 0x7fe30c1430c0 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.715+0000 7fe31432b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe30c06fcc0 con 0x7fe30c13f710 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.715+0000 7fe31432b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:32.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.715+0000 7fe31432b640 10 monclient(hunting): authenticate will time out at 7291.937724s 2024-03-25T18:21:32.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.725+0000 7fe3120c7640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fe30c0a98f0 0x7fe30c070910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:57876/0 (socket says 172.21.15.22:57876) 2024-03-25T18:21:32.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.725+0000 7fe3120c7640 1 -- 172.21.15.22:0/1277942665 learned_addr learned my addr 172.21.15.22:0/1277942665 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.727+0000 7fe302ffd640 10 client.?.objecter ms_handle_connect 0x7fe30c0a98f0 2024-03-25T18:21:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.727+0000 7fe302ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.729+0000 7fe302ffd640 10 client.?.objecter ms_handle_connect 0x7fe30c1430c0 2024-03-25T18:21:32.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.729+0000 7fe302ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.731+0000 7fe302ffd640 10 client.?.objecter ms_handle_connect 0x7fe30c13f710 2024-03-25T18:21:32.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.731+0000 7fe302ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.738+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366773057 0 0) 0x7fe30c06ef90 con 0x7fe30c0a98f0 2024-03-25T18:21:32.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.738+0000 7fe302ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.738+0000 7fe302ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.738+0000 7fe302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.738+0000 7fe302ffd640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:21:32.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.738+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f0004eb0 con 0x7fe30c0a98f0 2024-03-25T18:21:32.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.743+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053168222 0 0) 0x7fe30c06fcc0 con 0x7fe30c13f710 2024-03-25T18:21:32.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.743+0000 7fe302ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.743+0000 7fe302ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.743+0000 7fe302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.743+0000 7fe302ffd640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:21:32.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.743+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe30c06ef90 con 0x7fe30c13f710 2024-03-25T18:21:32.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.750+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621925828 0 0) 0x7fe30c06f9e0 con 0x7fe30c1430c0 2024-03-25T18:21:32.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.750+0000 7fe302ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.750+0000 7fe302ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.750+0000 7fe302ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.750+0000 7fe302ffd640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:21:32.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.750+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe30c06fcc0 con 0x7fe30c1430c0 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.753+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 982779378 0 0) 0x7fe2f0004eb0 con 0x7fe30c0a98f0 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.754+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe30c06f9e0 con 0x7fe30c0a98f0 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.754+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe308004080 con 0x7fe30c0a98f0 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.754+0000 7fe302ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.754+0000 7fe302ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.754+0000 7fe302ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.754 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.755 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.755 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:32.755 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:32.755 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:32.755 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:32.755 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:32.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.754+0000 7fe302ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:32.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.759+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3620402013 0 0) 0x7fe30c06ef90 con 0x7fe30c13f710 2024-03-25T18:21:32.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.759+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2f0004eb0 con 0x7fe30c13f710 2024-03-25T18:21:32.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.759+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe2fc001b40 con 0x7fe30c13f710 2024-03-25T18:21:32.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.759+0000 7fe302ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.759+0000 7fe302ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.759+0000 7fe302ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:32.760 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:32.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.773+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4173330610 0 0) 0x7fe30c06fcc0 con 0x7fe30c1430c0 2024-03-25T18:21:32.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.773+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe30c06ef90 con 0x7fe30c1430c0 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.773+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe304001b90 con 0x7fe30c1430c0 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.773+0000 7fe302ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.773+0000 7fe302ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.773+0000 7fe302ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:32.774 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656911882 0 0) 0x7fe30c06f9e0 con 0x7fe30c0a98f0 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 >> v1:172.21.15.22:6790/0 conn(0x7fe30c13f710 legacy=0x7fe30c071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 >> v1:172.21.15.33:6789/0 conn(0x7fe30c1430c0 legacy=0x7fe30c1e6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 1 monclient: found mon.a 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:21:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c06ffa0 con 0x7fe30c0a98f0 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.776+0000 7fe302ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 5 monclient: authenticate success, global_id 4323 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 10 monclient: _renew_subs 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c070280 con 0x7fe30c0a98f0 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 10 monclient: _renew_subs 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30c06e110 con 0x7fe30c0a98f0 2024-03-25T18:21:32.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.777+0000 7fe31432b640 1 librados: init done 2024-03-25T18:21:32.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.778+0000 7fe31432b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.778+0000 7fe31432b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.778+0000 7fe31432b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:21:32.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:32.778+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c14f2d0 con 0x7fe30c0a98f0 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.082+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe308000ed0 con 0x7fe30c0a98f0 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.082+0000 7fe302ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.082+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe3080012a0 con 0x7fe30c0a98f0 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.082+0000 7fe302ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.082+0000 7fe302ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.082+0000 7fe302ffd640 10 monclient: dump: 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:33.083 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:33.084 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:33.084 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:33.084 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:33.084 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:33.084 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:33.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.082+0000 7fe3108c4640 4 set_mon_vals no callback set 2024-03-25T18:21:33.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.083+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fe3080014c0 con 0x7fe30c0a98f0 2024-03-25T18:21:33.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.084+0000 7fe3118c6640 10 monclient: get_auth_request con 0x7fe2f006e0f0 auth_method 0 2024-03-25T18:21:33.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.084+0000 7fe302ffd640 10 client.4323.objecter ms_handle_connect 0x7fe2f006e0f0 2024-03-25T18:21:33.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fe3080547e0 con 0x7fe30c0a98f0 2024-03-25T18:21:33.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 10 client.4323.objecter ms_dispatch 0x7fe30c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:33.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:33.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:33.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T18:21:33.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3080502f0 con 0x7fe30c0a98f0 2024-03-25T18:21:33.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:33.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.090+0000 7fe302ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:33.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.320+0000 7fe31432b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.320+0000 7fe31432b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.320+0000 7fe31432b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:21:33.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.320+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe30c14d5a0 con 0x7fe30c0a98f0 2024-03-25T18:21:33.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.380+0000 7fe302ffd640 1 -- 172.21.15.22:0/1277942665 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fe308052220 con 0x7fe30c0a98f0 2024-03-25T18:21:33.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.380+0000 7fe302ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.380+0000 7fe302ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.382+0000 7fe31432b640 10 librados: watch_flush enter 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.382+0000 7fe31432b640 10 librados: watch_flush exit 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.382+0000 7fe31432b640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.382+0000 7fe31432b640 10 client.4323.objecter successfully canceled tick 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 >> v1:172.21.15.33:6800/215350958 conn(0x7fe2f006e0f0 legacy=0x7fe2f00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 10 monclient: shutdown 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 >> v1:172.21.15.22:6789/0 conn(0x7fe30c0a98f0 legacy=0x7fe30c070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 shutdown_connections 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 >> 172.21.15.22:0/1277942665 conn(0x7fe30c0b7ea0 msgr2=0x7fe30c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 shutdown_connections 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 1 -- 172.21.15.22:0/1277942665 wait complete. 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:33.383+0000 7fe31432b640 1 librados: shutdown 2024-03-25T18:21:33.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:40.201 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:40.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.317+0000 7febcc0ba640 10 monclient: get_monmap_and_config 2024-03-25T18:21:40.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.317+0000 7febcc0ba640 10 monclient: build_initial_monmap 2024-03-25T18:21:40.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.317+0000 7febcc0ba640 10 monclient: monmap: 2024-03-25T18:21:40.317 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:40.317 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:40.317634+0000 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:40.317634+0000 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:40.318 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.317+0000 7febcc0ba640 1 Processor -- start 2024-03-25T18:21:40.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.318+0000 7febcc0ba640 1 -- start start 2024-03-25T18:21:40.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.318+0000 7febcc0ba640 10 monclient: init 2024-03-25T18:21:40.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.318+0000 7febcc0ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:40.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.318+0000 7febcc0ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.318+0000 7febcc0ba640 10 monclient(hunting): picked mon.noname-b con 0x7febc40a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.318+0000 7febcc0ba640 10 monclient(hunting): picked mon.noname-c con 0x7febc413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.318+0000 7febcc0ba640 10 monclient(hunting): picked mon.noname-a con 0x7febc41430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.319+0000 7febcc0ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc4142520 con 0x7febc41430c0 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.319+0000 7febcc0ba640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40b28a0 con 0x7febc413f710 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.319+0000 7febcc0ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40b2a30 con 0x7febc40a98f0 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.319+0000 7febcc0ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:40.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.319+0000 7febcc0ba640 10 monclient(hunting): authenticate will time out at 7299.541627s 2024-03-25T18:21:40.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.327+0000 7febc9655640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7febc413f710 0x7febc41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:57048/0 (socket says 172.21.15.22:57048) 2024-03-25T18:21:40.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.327+0000 7febc9655640 1 -- 172.21.15.22:0/2075493034 learned_addr learned my addr 172.21.15.22:0/2075493034 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:40.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.340+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461043650 0 0) 0x7febc4142520 con 0x7febc41430c0 2024-03-25T18:21:40.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.340+0000 7febc8e54640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.340+0000 7febc8e54640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.340+0000 7febc8e54640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.340+0000 7febc8e54640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.340+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb0003520 con 0x7febc41430c0 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.341+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790596676 0 0) 0x7febc40b28a0 con 0x7febc413f710 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.341+0000 7febc8e54640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.341+0000 7febc8e54640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.341+0000 7febc8e54640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.341+0000 7febc8e54640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:21:40.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.341+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc4142520 con 0x7febc413f710 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2444043564 0 0) 0x7febc4142520 con 0x7febc413f710 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 >> v1:172.21.15.22:6790/0 conn(0x7febc40a98f0 legacy=0x7febc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 >> v1:172.21.15.22:6789/0 conn(0x7febc41430c0 legacy=0x7febc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 10 monclient: _finish_hunting 0 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 1 monclient: found mon.noname-c 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:21:40.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40b44c0 con 0x7febc413f710 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 10 monclient: _finish_auth 0 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 10 monclient: _check_auth_tickets 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7febb4001670 con 0x7febc413f710 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febcc0ba640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febcc0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febc8e54640 10 monclient: dump: 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:40.355 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:40.356 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:40.356 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:40.356 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.353+0000 7febcc0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:40.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febb40017d0 con 0x7febc413f710 2024-03-25T18:21:40.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febc8e54640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:40.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febc8e54640 1 -- 172.21.15.22:0/2075493034 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7febb4001ba0 con 0x7febc413f710 2024-03-25T18:21:40.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febcc0ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febcc0ba640 4 set_mon_vals no callback set 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febc8e54640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febc8e54640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.371+0000 7febc8e54640 10 monclient: dump: 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:40.372 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.372+0000 7febcc0ba640 10 monclient: shutdown 2024-03-25T18:21:40.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.372+0000 7febcc0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:40.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.372+0000 7febcc0ba640 1 -- 172.21.15.22:0/2075493034 >> v1:172.21.15.33:6789/0 conn(0x7febc413f710 legacy=0x7febc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.372+0000 7febcc0ba640 1 -- 172.21.15.22:0/2075493034 shutdown_connections 2024-03-25T18:21:40.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.372+0000 7febcc0ba640 1 -- 172.21.15.22:0/2075493034 >> 172.21.15.22:0/2075493034 conn(0x7febc40b7ea0 msgr2=0x7febc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:40.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.372+0000 7febcc0ba640 1 -- 172.21.15.22:0/2075493034 shutdown_connections 2024-03-25T18:21:40.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.372+0000 7febcc0ba640 1 -- 172.21.15.22:0/2075493034 wait complete. 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.373+0000 7febcc0ba640 10 monclient: build_initial_monmap 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.373+0000 7febcc0ba640 10 monclient: monmap: 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:40.374 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:40.375 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:40.375 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:40.375 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.374+0000 7febcc0ba640 1 librados: starting msgr at 2024-03-25T18:21:40.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.374+0000 7febcc0ba640 1 librados: starting objecter 2024-03-25T18:21:40.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.374+0000 7febcc0ba640 1 Processor -- start 2024-03-25T18:21:40.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.374+0000 7febcc0ba640 1 -- start start 2024-03-25T18:21:40.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.374+0000 7febcc0ba640 1 librados: setting wanted keys 2024-03-25T18:21:40.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.374+0000 7febcc0ba640 1 librados: calling monclient init 2024-03-25T18:21:40.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.374+0000 7febcc0ba640 10 monclient: init 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 10 monclient(hunting): picked mon.noname-c con 0x7febc40a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 10 monclient(hunting): picked mon.noname-a con 0x7febc413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 10 monclient(hunting): picked mon.noname-b con 0x7febc41430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc41db660 con 0x7febc413f710 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc41db7f0 con 0x7febc41430c0 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc414dee0 con 0x7febc40a98f0 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:40.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.375+0000 7febcc0ba640 10 monclient(hunting): authenticate will time out at 7299.598126s 2024-03-25T18:21:40.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.387+0000 7febc9655640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7febc413f710 0x7febc40ae760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:33506/0 (socket says 172.21.15.22:33506) 2024-03-25T18:21:40.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.387+0000 7febc9655640 1 -- 172.21.15.22:0/290952087 learned_addr learned my addr 172.21.15.22:0/290952087 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:40.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.389+0000 7febba7fc640 10 client.?.objecter ms_handle_connect 0x7febc41430c0 2024-03-25T18:21:40.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.389+0000 7febba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.390+0000 7febba7fc640 10 client.?.objecter ms_handle_connect 0x7febc40a98f0 2024-03-25T18:21:40.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.390+0000 7febba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.391+0000 7febba7fc640 10 client.?.objecter ms_handle_connect 0x7febc413f710 2024-03-25T18:21:40.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.391+0000 7febba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868161194 0 0) 0x7febc41db660 con 0x7febc413f710 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba8004eb0 con 0x7febc413f710 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914712029 0 0) 0x7febc414dee0 con 0x7febc40a98f0 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:21:40.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.396+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc41db660 con 0x7febc40a98f0 2024-03-25T18:21:40.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.401+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478847804 0 0) 0x7febc41db7f0 con 0x7febc41430c0 2024-03-25T18:21:40.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.401+0000 7febba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.401+0000 7febba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.401+0000 7febba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.401+0000 7febba7fc640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:21:40.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.401+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc414dee0 con 0x7febc41430c0 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.407+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851529401 0 0) 0x7feba8004eb0 con 0x7febc413f710 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.407+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc41db7f0 con 0x7febc413f710 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.407+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7febb4001ba0 con 0x7febc413f710 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.407+0000 7febba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.407+0000 7febba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.407+0000 7febba7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.407+0000 7febba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.417+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2676890643 0 0) 0x7febc414dee0 con 0x7febc41430c0 2024-03-25T18:21:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.417+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feba8004eb0 con 0x7febc41430c0 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.417+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7febbc001c10 con 0x7febc41430c0 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.417+0000 7febba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.417+0000 7febba7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.417+0000 7febba7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:40.418 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.422+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 946203511 0 0) 0x7febc41db660 con 0x7febc40a98f0 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.422+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc414dee0 con 0x7febc40a98f0 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.422+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7febc00018a0 con 0x7febc40a98f0 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.422+0000 7febba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.422+0000 7febba7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.422+0000 7febba7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:40.423 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 841411356 0 0) 0x7feba8004eb0 con 0x7febc41430c0 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 >> v1:172.21.15.22:6790/0 conn(0x7febc40a98f0 legacy=0x7febc41dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 >> v1:172.21.15.22:6789/0 conn(0x7febc413f710 legacy=0x7febc40ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 1 monclient: found mon.b 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:40.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc414e070 con 0x7febc41430c0 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.427+0000 7febba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 10 monclient: _renew_subs 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc40b11e0 con 0x7febc41430c0 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 10 monclient: _renew_subs 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febc41e3470 con 0x7febc41430c0 2024-03-25T18:21:40.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.428+0000 7febcc0ba640 1 librados: init done 2024-03-25T18:21:40.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.429+0000 7febcc0ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:40.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.429+0000 7febcc0ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:40.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.429+0000 7febcc0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:40.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.429+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc414e260 con 0x7febc41430c0 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.759+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febbc001310 con 0x7febc41430c0 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.759+0000 7febba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.759+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7febbc0020e0 con 0x7febc41430c0 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.759+0000 7febba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.759+0000 7febba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.759+0000 7febba7fc640 10 monclient: dump: 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:40.759 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:40.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.759+0000 7febbbfff640 4 set_mon_vals no callback set 2024-03-25T18:21:40.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.760+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7febbc019a20 con 0x7febc41430c0 2024-03-25T18:21:40.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.760+0000 7febc9e56640 10 monclient: get_auth_request con 0x7feba806e0c0 auth_method 0 2024-03-25T18:21:40.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.760+0000 7febba7fc640 10 client.4330.objecter ms_handle_connect 0x7feba806e0c0 2024-03-25T18:21:40.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.766+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7febbc0537b0 con 0x7febc41430c0 2024-03-25T18:21:40.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.767+0000 7febba7fc640 10 client.4330.objecter ms_dispatch 0x7febc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:40.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.767+0000 7febba7fc640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:40.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.767+0000 7febba7fc640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:40.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.767+0000 7febba7fc640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T18:21:40.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.767+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febbc04f0f0 con 0x7febc41430c0 2024-03-25T18:21:40.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.767+0000 7febba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:40.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.767+0000 7febba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:40.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.990+0000 7febcc0ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.990+0000 7febcc0ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.990+0000 7febcc0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:40.990+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febc41db380 con 0x7febc41430c0 2024-03-25T18:21:41.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.066+0000 7febba7fc640 1 -- 172.21.15.22:0/290952087 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7febbc0511f0 con 0x7febc41430c0 2024-03-25T18:21:41.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.066+0000 7febba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:41.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.066+0000 7febba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:41.068 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:41.068 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 10 librados: watch_flush enter 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 10 librados: watch_flush exit 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 10 client.4330.objecter successfully canceled tick 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 >> v1:172.21.15.33:6800/215350958 conn(0x7feba806e0c0 legacy=0x7feba8070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 10 monclient: shutdown 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 >> v1:172.21.15.33:6789/0 conn(0x7febc41430c0 legacy=0x7febc41e1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 shutdown_connections 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 >> 172.21.15.22:0/290952087 conn(0x7febc40b7ea0 msgr2=0x7febc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 shutdown_connections 2024-03-25T18:21:41.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 1 -- 172.21.15.22:0/290952087 wait complete. 2024-03-25T18:21:41.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:41.069+0000 7febcc0ba640 1 librados: shutdown 2024-03-25T18:21:41.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:47.888 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.002+0000 7fc7fefc6640 10 monclient: get_monmap_and_config 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.002+0000 7fc7fefc6640 10 monclient: build_initial_monmap 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.002+0000 7fc7fefc6640 10 monclient: monmap: 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:48.003598+0000 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:48.003598+0000 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:48.003 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:48.004 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:48.004 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:48.004 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:48.004 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:48.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.003+0000 7fc7fefc6640 1 Processor -- start 2024-03-25T18:21:48.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.003+0000 7fc7fefc6640 1 -- start start 2024-03-25T18:21:48.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.003+0000 7fc7fefc6640 10 monclient: init 2024-03-25T18:21:48.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.003+0000 7fc7fefc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:48.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.003+0000 7fc7fefc6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:48.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.003+0000 7fc7fefc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f80b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:48.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.004+0000 7fc7fefc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f80b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:48.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.004+0000 7fc7fefc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f814be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:48.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.004+0000 7fc7fefc6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f80af940 con 0x7fc7f814be90 2024-03-25T18:21:48.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.004+0000 7fc7fefc6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f80b0a70 con 0x7fc7f80b1730 2024-03-25T18:21:48.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.004+0000 7fc7fefc6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f80b0d50 con 0x7fc7f80b4d60 2024-03-25T18:21:48.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.004+0000 7fc7fefc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:48.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.004+0000 7fc7fefc6640 10 monclient(hunting): authenticate will time out at 7307.227850s 2024-03-25T18:21:48.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.009+0000 7fc7effff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fc7f80b1730 0x7fc7f80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:44758/0 (socket says 172.21.15.22:44758) 2024-03-25T18:21:48.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.010+0000 7fc7effff640 1 -- 172.21.15.22:0/1356393928 learned_addr learned my addr 172.21.15.22:0/1356393928 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:48.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.030+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985559048 0 0) 0x7fc7f80b0a70 con 0x7fc7f80b1730 2024-03-25T18:21:48.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.030+0000 7fc7ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.030+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.030+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.030+0000 7fc7ef7fe640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:21:48.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.030+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e0003520 con 0x7fc7f80b1730 2024-03-25T18:21:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.038+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289595049 0 0) 0x7fc7f80b0d50 con 0x7fc7f80b4d60 2024-03-25T18:21:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.038+0000 7fc7ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.038+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.038+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.038+0000 7fc7ef7fe640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:21:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.038+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f80b0a70 con 0x7fc7f80b4d60 2024-03-25T18:21:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.039+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033956105 0 0) 0x7fc7f80af940 con 0x7fc7f814be90 2024-03-25T18:21:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.039+0000 7fc7ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.039+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.039+0000 7fc7ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.039+0000 7fc7ef7fe640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:21:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.039+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f80b0d50 con 0x7fc7f814be90 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 844915259 0 0) 0x7fc7e0003520 con 0x7fc7f80b1730 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 >> v1:172.21.15.22:6790/0 conn(0x7fc7f80b4d60 legacy=0x7fc7f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 >> v1:172.21.15.22:6789/0 conn(0x7fc7f814be90 legacy=0x7fc7f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 1 monclient: found mon.noname-c 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f80b1030 con 0x7fc7f80b1730 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:48.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc7e8001690 con 0x7fc7f80b1730 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7ef7fe640 10 monclient: dump: 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7fefc6640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:21:48.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.052+0000 7fc7fefc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7e80017f0 con 0x7fc7f80b1730 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7ef7fe640 1 -- 172.21.15.22:0/1356393928 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc7e8001c00 con 0x7fc7f80b1730 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7ef7fe640 10 monclient: dump: 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:48.067 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:48.068 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:48.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7fefc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:48.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.065+0000 7fc7fefc6640 4 set_mon_vals no callback set 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.066+0000 7fc7fefc6640 10 monclient: shutdown 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.066+0000 7fc7fefc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.066+0000 7fc7fefc6640 1 -- 172.21.15.22:0/1356393928 >> v1:172.21.15.33:6789/0 conn(0x7fc7f80b1730 legacy=0x7fc7f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.066+0000 7fc7fefc6640 1 -- 172.21.15.22:0/1356393928 shutdown_connections 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.066+0000 7fc7fefc6640 1 -- 172.21.15.22:0/1356393928 >> 172.21.15.22:0/1356393928 conn(0x7fc7f80617a0 msgr2=0x7fc7f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.066+0000 7fc7fefc6640 1 -- 172.21.15.22:0/1356393928 shutdown_connections 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.066+0000 7fc7fefc6640 1 -- 172.21.15.22:0/1356393928 wait complete. 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.067+0000 7fc7fefc6640 10 monclient: build_initial_monmap 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.067+0000 7fc7fefc6640 10 monclient: monmap: 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:48.069 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.068+0000 7fc7fefc6640 1 librados: starting msgr at 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.068+0000 7fc7fefc6640 1 librados: starting objecter 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.068+0000 7fc7fefc6640 1 Processor -- start 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.068+0000 7fc7fefc6640 1 -- start start 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.068+0000 7fc7fefc6640 1 librados: setting wanted keys 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.068+0000 7fc7fefc6640 1 librados: calling monclient init 2024-03-25T18:21:48.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.068+0000 7fc7fefc6640 10 monclient: init 2024-03-25T18:21:48.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f80b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f80b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f814be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f813bfb0 con 0x7fc7f80b4d60 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f81458e0 con 0x7fc7f814be90 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f812f7b0 con 0x7fc7f80b1730 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:48.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.069+0000 7fc7fefc6640 10 monclient(hunting): authenticate will time out at 7307.293094s 2024-03-25T18:21:48.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.078+0000 7fc7fcd62640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc7f80b1730 0x7fc7f813b8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47562/0 (socket says 172.21.15.22:47562) 2024-03-25T18:21:48.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.078+0000 7fc7fcd62640 1 -- 172.21.15.22:0/931521466 learned_addr learned my addr 172.21.15.22:0/931521466 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:48.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.079+0000 7fc7ed7fa640 10 client.?.objecter ms_handle_connect 0x7fc7f814be90 2024-03-25T18:21:48.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.079+0000 7fc7ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:48.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.081+0000 7fc7ed7fa640 10 client.?.objecter ms_handle_connect 0x7fc7f80b1730 2024-03-25T18:21:48.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.081+0000 7fc7ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:48.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.084+0000 7fc7ed7fa640 10 client.?.objecter ms_handle_connect 0x7fc7f80b4d60 2024-03-25T18:21:48.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.084+0000 7fc7ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.095+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4258982534 0 0) 0x7fc7f812f7b0 con 0x7fc7f80b1730 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.095+0000 7fc7ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.095+0000 7fc7ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.095+0000 7fc7ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.095+0000 7fc7ed7fa640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.095+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7e4004eb0 con 0x7fc7f80b1730 2024-03-25T18:21:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.100+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177054270 0 0) 0x7fc7f81458e0 con 0x7fc7f814be90 2024-03-25T18:21:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.100+0000 7fc7ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.100+0000 7fc7ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.100+0000 7fc7ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.100+0000 7fc7ed7fa640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:21:48.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.100+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f812f7b0 con 0x7fc7f814be90 2024-03-25T18:21:48.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.103+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374041159 0 0) 0x7fc7f813bfb0 con 0x7fc7f80b4d60 2024-03-25T18:21:48.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.103+0000 7fc7ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.103+0000 7fc7ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.103+0000 7fc7ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.103+0000 7fc7ed7fa640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:21:48.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.103+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f81458e0 con 0x7fc7f80b4d60 2024-03-25T18:21:48.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.117+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 376541602 0 0) 0x7fc7e4004eb0 con 0x7fc7f80b1730 2024-03-25T18:21:48.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.117+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7f813bfb0 con 0x7fc7f80b1730 2024-03-25T18:21:48.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.117+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc7f4001db0 con 0x7fc7f80b1730 2024-03-25T18:21:48.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.117+0000 7fc7ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.117+0000 7fc7ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.117+0000 7fc7ed7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:48.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.117+0000 7fc7ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.121+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882455618 0 0) 0x7fc7f812f7b0 con 0x7fc7f814be90 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.121+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7e4004eb0 con 0x7fc7f814be90 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.121+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc7f00017d0 con 0x7fc7f814be90 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.121+0000 7fc7ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.121+0000 7fc7ed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.121+0000 7fc7ed7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:48.123 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:48.124 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:48.124 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 776676964 0 0) 0x7fc7f813bfb0 con 0x7fc7f80b1730 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 >> v1:172.21.15.33:6789/0 conn(0x7fc7f814be90 legacy=0x7fc7f812f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 >> v1:172.21.15.22:6789/0 conn(0x7fc7f80b4d60 legacy=0x7fc7f812e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 1 monclient: found mon.c 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f813aa90 con 0x7fc7f80b1730 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:21:48.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 10 monclient: _renew_subs 2024-03-25T18:21:48.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:48.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f813ad70 con 0x7fc7f80b1730 2024-03-25T18:21:48.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:48.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 10 monclient: _renew_subs 2024-03-25T18:21:48.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:48.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f8145330 con 0x7fc7f80b1730 2024-03-25T18:21:48.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.128+0000 7fc7fefc6640 1 librados: init done 2024-03-25T18:21:48.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.129+0000 7fc7fefc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:48.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.129+0000 7fc7fefc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:48.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.129+0000 7fc7fefc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:48.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.129+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f813b580 con 0x7fc7f80b1730 2024-03-25T18:21:48.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.428+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7f4001ff0 con 0x7fc7f80b1730 2024-03-25T18:21:48.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.428+0000 7fc7ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:48.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.428+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fc7f4003de0 con 0x7fc7f80b1730 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.428+0000 7fc7ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.428+0000 7fc7ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.428+0000 7fc7ed7fa640 10 monclient: dump: 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:48.430 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:48.431 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:48.431 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:48.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.428+0000 7fc7eeffd640 4 set_mon_vals no callback set 2024-03-25T18:21:48.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.429+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fc7f4003fc0 con 0x7fc7f80b1730 2024-03-25T18:21:48.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.429+0000 7fc7effff640 10 monclient: get_auth_request con 0x7fc7e406df70 auth_method 0 2024-03-25T18:21:48.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.430+0000 7fc7ed7fa640 10 client.4337.objecter ms_handle_connect 0x7fc7e406df70 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fc7f4054bb0 con 0x7fc7f80b1730 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 10 client.4337.objecter ms_dispatch 0x7fc7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7f40506c0 con 0x7fc7f80b1730 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:48.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.434+0000 7fc7ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:48.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.667+0000 7fc7fefc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:48.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.667+0000 7fc7fefc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:48.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.667+0000 7fc7fefc6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:48.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.667+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7f81401f0 con 0x7fc7f80b1730 2024-03-25T18:21:48.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.710+0000 7fc7ed7fa640 1 -- 172.21.15.22:0/931521466 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fc7f40525f0 con 0x7fc7f80b1730 2024-03-25T18:21:48.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.710+0000 7fc7ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:48.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.710+0000 7fc7ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:48.711 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:48.711 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:48.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 10 librados: watch_flush enter 2024-03-25T18:21:48.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 10 librados: watch_flush exit 2024-03-25T18:21:48.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 10 client.4337.objecter successfully canceled tick 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 >> v1:172.21.15.33:6800/215350958 conn(0x7fc7e406df70 legacy=0x7fc7e4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 10 monclient: shutdown 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 >> v1:172.21.15.22:6790/0 conn(0x7fc7f80b1730 legacy=0x7fc7f813b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 shutdown_connections 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 >> 172.21.15.22:0/931521466 conn(0x7fc7f80617a0 msgr2=0x7fc7f813c730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.712+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 shutdown_connections 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.713+0000 7fc7fefc6640 1 -- 172.21.15.22:0/931521466 wait complete. 2024-03-25T18:21:48.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:48.713+0000 7fc7fefc6640 1 librados: shutdown 2024-03-25T18:21:48.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:52.355 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:55.017 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.373 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:55.035 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.406 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:02:55.068 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.409 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:55.072 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.423 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:55.041 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.425 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:55.087 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.432 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:55.050 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.451 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:02:55.113 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:52.454 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:02:55.072 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.532 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.645+0000 7f4112944640 10 monclient: get_monmap_and_config 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.645+0000 7f4112944640 10 monclient: build_initial_monmap 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.645+0000 7f4112944640 10 monclient: monmap: 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:21:55.647224+0000 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:21:55.647224+0000 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:55.647 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:55.648 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:55.648 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:21:55.648 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:21:55.648 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.646+0000 7f4112944640 1 Processor -- start 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.646+0000 7f4112944640 1 -- start start 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.646+0000 7f4112944640 10 monclient: init 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 10 monclient(hunting): picked mon.noname-a con 0x7f410c0b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 10 monclient(hunting): picked mon.noname-c con 0x7f410c0b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 10 monclient(hunting): picked mon.noname-b con 0x7f410c14be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:55.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0af940 con 0x7f410c0b4d60 2024-03-25T18:21:55.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0b0a70 con 0x7f410c0b1730 2024-03-25T18:21:55.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c0b0d50 con 0x7f410c14be90 2024-03-25T18:21:55.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:55.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.647+0000 7f4112944640 10 monclient(hunting): authenticate will time out at 7314.871236s 2024-03-25T18:21:55.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.658+0000 7f4103fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f410c0b1730 0x7f410c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:38040/0 (socket says 172.21.15.22:38040) 2024-03-25T18:21:55.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.658+0000 7f4103fff640 1 -- 172.21.15.22:0/2998719564 learned_addr learned my addr 172.21.15.22:0/2998719564 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:55.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.667+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011184319 0 0) 0x7f410c0b0d50 con 0x7f410c14be90 2024-03-25T18:21:55.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.667+0000 7f410b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.667+0000 7f410b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.667+0000 7f410b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.667+0000 7f410b7fe640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:21:55.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.667+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f8003520 con 0x7f410c14be90 2024-03-25T18:21:55.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.672+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879059402 0 0) 0x7f410c0b0a70 con 0x7f410c0b1730 2024-03-25T18:21:55.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.672+0000 7f410b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.672+0000 7f410b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.672+0000 7f410b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.672+0000 7f410b7fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:21:55.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.672+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c0b0d50 con 0x7f410c0b1730 2024-03-25T18:21:55.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.677+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039572338 0 0) 0x7f410c0af940 con 0x7f410c0b4d60 2024-03-25T18:21:55.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.677+0000 7f410b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.677+0000 7f410b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.677+0000 7f410b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.677+0000 7f410b7fe640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:21:55.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.677+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c0b0a70 con 0x7f410c0b4d60 2024-03-25T18:21:55.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4175570810 0 0) 0x7f410c0b0d50 con 0x7f410c0b1730 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 >> v1:172.21.15.22:6790/0 conn(0x7f410c14be90 legacy=0x7f410c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 >> v1:172.21.15.22:6789/0 conn(0x7f410c0b4d60 legacy=0x7f410c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c0b1030 con 0x7f410c0b1730 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f40f4001670 con 0x7f410c0b1730 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f4112944640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f4112944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f410b7fe640 10 monclient: dump: 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:55.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.684+0000 7f4112944640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40f40017d0 con 0x7f410c0b1730 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f410b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f410b7fe640 1 -- 172.21.15.22:0/2998719564 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f40f4001be0 con 0x7f410c0b1730 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 4 set_mon_vals no callback set 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f410b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f410b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f410b7fe640 10 monclient: dump: 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:55.697 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:55.698 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:55.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 10 monclient: shutdown 2024-03-25T18:21:55.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 1 -- 172.21.15.22:0/2998719564 >> v1:172.21.15.33:6789/0 conn(0x7f410c0b1730 legacy=0x7f410c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 1 -- 172.21.15.22:0/2998719564 shutdown_connections 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 1 -- 172.21.15.22:0/2998719564 >> 172.21.15.22:0/2998719564 conn(0x7f410c0617a0 msgr2=0x7f410c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 1 -- 172.21.15.22:0/2998719564 shutdown_connections 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.696+0000 7f4112944640 1 -- 172.21.15.22:0/2998719564 wait complete. 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.697+0000 7f4112944640 10 monclient: build_initial_monmap 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.697+0000 7f4112944640 10 monclient: monmap: 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:21:55.700 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.698+0000 7f4112944640 1 librados: starting msgr at 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.698+0000 7f4112944640 1 librados: starting objecter 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.698+0000 7f4112944640 1 Processor -- start 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.698+0000 7f4112944640 1 -- start start 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.698+0000 7f4112944640 1 librados: setting wanted keys 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.698+0000 7f4112944640 1 librados: calling monclient init 2024-03-25T18:21:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.698+0000 7f4112944640 10 monclient: init 2024-03-25T18:21:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 10 monclient(hunting): picked mon.noname-a con 0x7f410c0b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:21:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 10 monclient(hunting): picked mon.noname-b con 0x7f410c0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:21:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 10 monclient(hunting): picked mon.noname-c con 0x7f410c14be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:21:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c140720 con 0x7f410c0b1730 2024-03-25T18:21:55.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c140380 con 0x7f410c0b4d60 2024-03-25T18:21:55.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410c13af80 con 0x7f410c14be90 2024-03-25T18:21:55.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:55.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.699+0000 7f4112944640 10 monclient(hunting): authenticate will time out at 7314.923060s 2024-03-25T18:21:55.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.705+0000 7f4110ee1640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f410c14be90 0x7f410c1db340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45632/0 (socket says 172.21.15.22:45632) 2024-03-25T18:21:55.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.705+0000 7f4110ee1640 1 -- 172.21.15.22:0/2660913515 learned_addr learned my addr 172.21.15.22:0/2660913515 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:21:55.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.709+0000 7f41097fa640 10 client.?.objecter ms_handle_connect 0x7f410c14be90 2024-03-25T18:21:55.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.709+0000 7f41097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:55.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.715+0000 7f41097fa640 10 client.?.objecter ms_handle_connect 0x7f410c0b4d60 2024-03-25T18:21:55.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.715+0000 7f41097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:55.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.719+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770361425 0 0) 0x7f410c13af80 con 0x7f410c14be90 2024-03-25T18:21:55.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.719+0000 7f41097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.719+0000 7f41097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.719+0000 7f41097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.719+0000 7f41097fa640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:21:55.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.719+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ec004eb0 con 0x7f410c14be90 2024-03-25T18:21:55.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.729+0000 7f41097fa640 10 client.?.objecter ms_handle_connect 0x7f410c0b1730 2024-03-25T18:21:55.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.729+0000 7f41097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.730+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 92662342 0 0) 0x7f40ec004eb0 con 0x7f410c14be90 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.730+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f410c13af80 con 0x7f410c14be90 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.730+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4104001c30 con 0x7f410c14be90 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.730+0000 7f41097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.730+0000 7f41097fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.730+0000 7f41097fa640 10 monclient(hunting): dump: 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:55.731 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:55.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.730+0000 7f41097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:55.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.731+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308438185 0 0) 0x7f410c140380 con 0x7f410c0b4d60 2024-03-25T18:21:55.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.731+0000 7f41097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.731+0000 7f41097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.731+0000 7f41097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.731+0000 7f41097fa640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:21:55.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.731+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ec004eb0 con 0x7f410c0b4d60 2024-03-25T18:21:55.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.732+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050387063 0 0) 0x7f410c140720 con 0x7f410c0b1730 2024-03-25T18:21:55.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.732+0000 7f41097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.732+0000 7f41097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.732+0000 7f41097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.732+0000 7f41097fa640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:21:55.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.732+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f410c140380 con 0x7f410c0b1730 2024-03-25T18:21:55.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 622717207 0 0) 0x7f410c13af80 con 0x7f410c14be90 2024-03-25T18:21:55.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 >> v1:172.21.15.33:6789/0 conn(0x7f410c0b4d60 legacy=0x7f410c1dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 >> v1:172.21.15.22:6789/0 conn(0x7f410c0b1730 legacy=0x7f410c134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 1 monclient: found mon.c 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410c13b110 con 0x7f410c14be90 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f41097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 5 monclient: authenticate success, global_id 4343 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 10 monclient: _renew_subs 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410c13bab0 con 0x7f410c14be90 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:55.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 10 monclient: _renew_subs 2024-03-25T18:21:55.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:55.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f410c1356a0 con 0x7f410c14be90 2024-03-25T18:21:55.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.735+0000 7f4112944640 1 librados: init done 2024-03-25T18:21:55.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.736+0000 7f4112944640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:55.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.736+0000 7f4112944640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:55.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.736+0000 7f4112944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:55.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:55.736+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f410c1401f0 con 0x7f410c14be90 2024-03-25T18:21:56.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.067+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4104003970 con 0x7f410c14be90 2024-03-25T18:21:56.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.067+0000 7f41097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:56.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.067+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4104003d40 con 0x7f410c14be90 2024-03-25T18:21:56.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.067+0000 7f41097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.067+0000 7f41097fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.067+0000 7f41097fa640 10 monclient: dump: 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:21:56.069 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:21:56.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.067+0000 7f410affd640 4 set_mon_vals no callback set 2024-03-25T18:21:56.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.068+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f4104003f60 con 0x7f410c14be90 2024-03-25T18:21:56.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.068+0000 7f410bfff640 10 monclient: get_auth_request con 0x7f40ec06de80 auth_method 0 2024-03-25T18:21:56.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.069+0000 7f41097fa640 10 client.4343.objecter ms_handle_connect 0x7f40ec06de80 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f41040549c0 con 0x7f410c14be90 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 10 client.4343.objecter ms_dispatch 0x7f410c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41040504d0 con 0x7f410c14be90 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.075+0000 7f41097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:56.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.322+0000 7f4112944640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.322+0000 7f4112944640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.322+0000 7f4112944640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:21:56.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.322+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f410c140510 con 0x7f410c14be90 2024-03-25T18:21:56.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.356+0000 7f41097fa640 1 -- 172.21.15.22:0/2660913515 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f4104052400 con 0x7f410c14be90 2024-03-25T18:21:56.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.356+0000 7f41097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.356+0000 7f41097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:56.358 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:21:56.358 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 10 librados: watch_flush enter 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 10 librados: watch_flush exit 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 10 client.4343.objecter successfully canceled tick 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 >> v1:172.21.15.33:6800/215350958 conn(0x7f40ec06de80 legacy=0x7f40ec070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 10 monclient: shutdown 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 >> v1:172.21.15.22:6790/0 conn(0x7f410c14be90 legacy=0x7f410c1db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 shutdown_connections 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.358+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 >> 172.21.15.22:0/2660913515 conn(0x7f410c0617a0 msgr2=0x7f410c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.359+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 shutdown_connections 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.359+0000 7f4112944640 1 -- 172.21.15.22:0/2660913515 wait complete. 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:21:56.359+0000 7f4112944640 1 librados: shutdown 2024-03-25T18:21:56.372 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:03.177 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:03.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.294+0000 7fbf2a948640 10 monclient: get_monmap_and_config 2024-03-25T18:22:03.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.294+0000 7fbf2a948640 10 monclient: build_initial_monmap 2024-03-25T18:22:03.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.294+0000 7fbf2a948640 10 monclient: monmap: 2024-03-25T18:22:03.296 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:03.296 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:03.296 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:03.296198+0000 2024-03-25T18:22:03.296 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:03.296198+0000 2024-03-25T18:22:03.297 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:03.297 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:03.297 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:03.297 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:03.297 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:03.297 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.295+0000 7fbf2a948640 1 Processor -- start 2024-03-25T18:22:03.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.295+0000 7fbf2a948640 1 -- start start 2024-03-25T18:22:03.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.295+0000 7fbf2a948640 10 monclient: init 2024-03-25T18:22:03.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:03.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:03.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 10 monclient(hunting): picked mon.noname-b con 0x7fbf240b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:03.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 10 monclient(hunting): picked mon.noname-c con 0x7fbf240b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:03.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 10 monclient(hunting): picked mon.noname-a con 0x7fbf2414be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:03.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240af940 con 0x7fbf2414be90 2024-03-25T18:22:03.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240b0a70 con 0x7fbf240b1730 2024-03-25T18:22:03.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240b0d50 con 0x7fbf240b4d60 2024-03-25T18:22:03.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:03.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.296+0000 7fbf2a948640 10 monclient(hunting): authenticate will time out at 7322.520158s 2024-03-25T18:22:03.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.303+0000 7fbf23fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fbf240b4d60 0x7fbf2413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49902/0 (socket says 172.21.15.22:49902) 2024-03-25T18:22:03.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.303+0000 7fbf23fff640 1 -- 172.21.15.22:0/945317189 learned_addr learned my addr 172.21.15.22:0/945317189 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:03.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.309+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497272362 0 0) 0x7fbf240b0d50 con 0x7fbf240b4d60 2024-03-25T18:22:03.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.310+0000 7fbf22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.310+0000 7fbf22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.310+0000 7fbf22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.310+0000 7fbf22ffd640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:22:03.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.310+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c003520 con 0x7fbf240b4d60 2024-03-25T18:22:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.341+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065427823 0 0) 0x7fbf240b0a70 con 0x7fbf240b1730 2024-03-25T18:22:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.341+0000 7fbf22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.341+0000 7fbf22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.341+0000 7fbf22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.341+0000 7fbf22ffd640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:22:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.341+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf240b0d50 con 0x7fbf240b1730 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 933043320 0 0) 0x7fbf0c003520 con 0x7fbf240b4d60 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 >> v1:172.21.15.33:6789/0 conn(0x7fbf240b1730 legacy=0x7fbf240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 >> v1:172.21.15.22:6789/0 conn(0x7fbf2414be90 legacy=0x7fbf2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 1 monclient: found mon.noname-b 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240b1030 con 0x7fbf240b4d60 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:03.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf14001690 con 0x7fbf240b4d60 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf22ffd640 10 monclient: dump: 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf2a948640 5 monclient: authenticate success, global_id 4346 2024-03-25T18:22:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.345+0000 7fbf2a948640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf140017f0 con 0x7fbf240b4d60 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf22ffd640 1 -- 172.21.15.22:0/945317189 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf14001c00 con 0x7fbf240b4d60 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf2a948640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf2a948640 4 set_mon_vals no callback set 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.359+0000 7fbf22ffd640 10 monclient: dump: 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.361 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.362 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:03.362 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:03.362 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:03.362 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:03.362 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.360+0000 7fbf2a948640 10 monclient: shutdown 2024-03-25T18:22:03.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.360+0000 7fbf2a948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:03.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.360+0000 7fbf2a948640 1 -- 172.21.15.22:0/945317189 >> v1:172.21.15.22:6790/0 conn(0x7fbf240b4d60 legacy=0x7fbf2413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.360+0000 7fbf2a948640 1 -- 172.21.15.22:0/945317189 shutdown_connections 2024-03-25T18:22:03.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.360+0000 7fbf2a948640 1 -- 172.21.15.22:0/945317189 >> 172.21.15.22:0/945317189 conn(0x7fbf240617a0 msgr2=0x7fbf24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.360+0000 7fbf2a948640 1 -- 172.21.15.22:0/945317189 shutdown_connections 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.360+0000 7fbf2a948640 1 -- 172.21.15.22:0/945317189 wait complete. 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.361+0000 7fbf2a948640 10 monclient: build_initial_monmap 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.361+0000 7fbf2a948640 10 monclient: monmap: 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.361+0000 7fbf2a948640 1 librados: starting msgr at 2024-03-25T18:22:03.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.361+0000 7fbf2a948640 1 librados: starting objecter 2024-03-25T18:22:03.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.361+0000 7fbf2a948640 1 Processor -- start 2024-03-25T18:22:03.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.362+0000 7fbf2a948640 1 -- start start 2024-03-25T18:22:03.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.362+0000 7fbf2a948640 1 librados: setting wanted keys 2024-03-25T18:22:03.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.362+0000 7fbf2a948640 1 librados: calling monclient init 2024-03-25T18:22:03.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.362+0000 7fbf2a948640 10 monclient: init 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.362+0000 7fbf2a948640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 10 monclient(hunting): picked mon.noname-c con 0x7fbf240b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 10 monclient(hunting): picked mon.noname-b con 0x7fbf240b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 10 monclient(hunting): picked mon.noname-a con 0x7fbf2414be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf241361d0 con 0x7fbf2414be90 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf24136c40 con 0x7fbf240b4d60 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf24136f20 con 0x7fbf240b1730 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:03.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.363+0000 7fbf2a948640 10 monclient(hunting): authenticate will time out at 7322.586746s 2024-03-25T18:22:03.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.371+0000 7fbf28ee5640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fbf2414be90 0x7fbf241e33d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:40530/0 (socket says 172.21.15.22:40530) 2024-03-25T18:22:03.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.371+0000 7fbf28ee5640 1 -- 172.21.15.22:0/1724485468 learned_addr learned my addr 172.21.15.22:0/1724485468 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:03.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.374+0000 7fbf20ff9640 10 client.?.objecter ms_handle_connect 0x7fbf2414be90 2024-03-25T18:22:03.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.374+0000 7fbf20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.376+0000 7fbf20ff9640 10 client.?.objecter ms_handle_connect 0x7fbf240b4d60 2024-03-25T18:22:03.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.376+0000 7fbf20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 10 client.?.objecter ms_handle_connect 0x7fbf240b1730 2024-03-25T18:22:03.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599750583 0 0) 0x7fbf24136c40 con 0x7fbf240b4d60 2024-03-25T18:22:03.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:22:03.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.382+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf10004eb0 con 0x7fbf240b4d60 2024-03-25T18:22:03.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.392+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464698097 0 0) 0x7fbf241361d0 con 0x7fbf2414be90 2024-03-25T18:22:03.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.392+0000 7fbf20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.392+0000 7fbf20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.392+0000 7fbf20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.392+0000 7fbf20ff9640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:22:03.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.392+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf24136c40 con 0x7fbf2414be90 2024-03-25T18:22:03.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.396+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453169973 0 0) 0x7fbf24136f20 con 0x7fbf240b1730 2024-03-25T18:22:03.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.396+0000 7fbf20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.396+0000 7fbf20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.396+0000 7fbf20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.396+0000 7fbf20ff9640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:22:03.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.396+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf241361d0 con 0x7fbf240b1730 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.408+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3006288194 0 0) 0x7fbf10004eb0 con 0x7fbf240b4d60 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.409+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf24136f20 con 0x7fbf240b4d60 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.409+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf18001970 con 0x7fbf240b4d60 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.409+0000 7fbf20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.409+0000 7fbf20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.409+0000 7fbf20ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.410 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.411 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.411 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:03.411 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:03.411 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:03.411 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:03.411 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.409+0000 7fbf20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:03.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.416+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3235354998 0 0) 0x7fbf24136c40 con 0x7fbf2414be90 2024-03-25T18:22:03.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.416+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf10004eb0 con 0x7fbf2414be90 2024-03-25T18:22:03.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.416+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf1c001bd0 con 0x7fbf2414be90 2024-03-25T18:22:03.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.416+0000 7fbf20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.416+0000 7fbf20ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.416+0000 7fbf20ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:03.418 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.417+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3873018879 0 0) 0x7fbf241361d0 con 0x7fbf240b1730 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.417+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf24136c40 con 0x7fbf240b1730 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.417+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf14001b00 con 0x7fbf240b1730 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.417+0000 7fbf20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.417+0000 7fbf20ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.417+0000 7fbf20ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:03.419 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:03.420 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:03.420 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:03.420 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1829701188 0 0) 0x7fbf24136f20 con 0x7fbf240b4d60 2024-03-25T18:22:03.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 >> v1:172.21.15.22:6790/0 conn(0x7fbf240b1730 legacy=0x7fbf2412e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 >> v1:172.21.15.22:6789/0 conn(0x7fbf2414be90 legacy=0x7fbf241e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:03.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 1 monclient: found mon.b 2024-03-25T18:22:03.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.420+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24137200 con 0x7fbf240b4d60 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf20ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf20ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 5 monclient: authenticate success, global_id 4348 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 10 monclient: _renew_subs 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf24135cb0 con 0x7fbf240b4d60 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 10 monclient: _renew_subs 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2413e930 con 0x7fbf240b4d60 2024-03-25T18:22:03.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 1 librados: init done 2024-03-25T18:22:03.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.421+0000 7fbf2a948640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.422+0000 7fbf2a948640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.422+0000 7fbf2a948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:03.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.422+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf241401f0 con 0x7fbf240b4d60 2024-03-25T18:22:03.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf18003610 con 0x7fbf240b4d60 2024-03-25T18:22:03.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf18003920 con 0x7fbf240b4d60 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf20ff9640 10 monclient: dump: 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:03.727 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:03.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf227fc640 4 set_mon_vals no callback set 2024-03-25T18:22:03.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.725+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fbf18019930 con 0x7fbf240b4d60 2024-03-25T18:22:03.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.726+0000 7fbf23fff640 10 monclient: get_auth_request con 0x7fbf1006e080 auth_method 0 2024-03-25T18:22:03.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.726+0000 7fbf20ff9640 10 client.4348.objecter ms_handle_connect 0x7fbf1006e080 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.730+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fbf180536b0 con 0x7fbf240b4d60 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.730+0000 7fbf20ff9640 10 client.4348.objecter ms_dispatch 0x7fbf24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.730+0000 7fbf20ff9640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.730+0000 7fbf20ff9640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.730+0000 7fbf20ff9640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.731+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf1804eff0 con 0x7fbf240b4d60 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.731+0000 7fbf20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.731+0000 7fbf20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:03.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.968+0000 7fbf2a948640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:03.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.968+0000 7fbf2a948640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:03.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.968+0000 7fbf2a948640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:03.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:03.968+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf240afbd0 con 0x7fbf240b4d60 2024-03-25T18:22:04.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.035+0000 7fbf20ff9640 1 -- 172.21.15.22:0/1724485468 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fbf180510f0 con 0x7fbf240b4d60 2024-03-25T18:22:04.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.035+0000 7fbf20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:04.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.035+0000 7fbf20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:04.037 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:04.037 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 10 librados: watch_flush enter 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 10 librados: watch_flush exit 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 10 client.4348.objecter successfully canceled tick 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 >> v1:172.21.15.33:6800/215350958 conn(0x7fbf1006e080 legacy=0x7fbf10070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 10 monclient: shutdown 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 >> v1:172.21.15.33:6789/0 conn(0x7fbf240b4d60 legacy=0x7fbf2412cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 shutdown_connections 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 >> 172.21.15.22:0/1724485468 conn(0x7fbf240617a0 msgr2=0x7fbf240b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 shutdown_connections 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 1 -- 172.21.15.22:0/1724485468 wait complete. 2024-03-25T18:22:04.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:04.038+0000 7fbf2a948640 1 librados: shutdown 2024-03-25T18:22:04.051 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:10.858 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.967+0000 7f9daef32640 10 monclient: get_monmap_and_config 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.967+0000 7f9daef32640 10 monclient: build_initial_monmap 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.967+0000 7f9daef32640 10 monclient: monmap: 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:10.968978+0000 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:10.968978+0000 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:10.969 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.968+0000 7f9daef32640 1 Processor -- start 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.968+0000 7f9daef32640 1 -- start start 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.968+0000 7f9daef32640 10 monclient: init 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 10 monclient(hunting): picked mon.noname-b con 0x7f9da80a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 10 monclient(hunting): picked mon.noname-c con 0x7f9da813f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 10 monclient(hunting): picked mon.noname-a con 0x7f9da81430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da8142520 con 0x7f9da81430c0 2024-03-25T18:22:10.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da80b28a0 con 0x7f9da813f710 2024-03-25T18:22:10.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da80b2a30 con 0x7f9da80a98f0 2024-03-25T18:22:10.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:10.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.969+0000 7f9daef32640 10 monclient(hunting): authenticate will time out at 7330.193279s 2024-03-25T18:22:10.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.978+0000 7f9da7fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f9da813f710 0x7f9da81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:49738/0 (socket says 172.21.15.22:49738) 2024-03-25T18:22:10.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:10.978+0000 7f9da7fff640 1 -- 172.21.15.22:0/1956351092 learned_addr learned my addr 172.21.15.22:0/1956351092 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:11.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834474320 0 0) 0x7f9da8142520 con 0x7f9da81430c0 2024-03-25T18:22:11.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d98003520 con 0x7f9da81430c0 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925207362 0 0) 0x7f9da80b28a0 con 0x7f9da813f710 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:22:11.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.004+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da8142520 con 0x7f9da813f710 2024-03-25T18:22:11.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.008+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179341551 0 0) 0x7f9da80b2a30 con 0x7f9da80a98f0 2024-03-25T18:22:11.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.008+0000 7f9da77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.008+0000 7f9da77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.008+0000 7f9da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.008+0000 7f9da77fe640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:22:11.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.008+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da80b28a0 con 0x7f9da80a98f0 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2735055852 0 0) 0x7f9da80b28a0 con 0x7f9da80a98f0 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 >> v1:172.21.15.33:6789/0 conn(0x7f9da813f710 legacy=0x7f9da81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 >> v1:172.21.15.22:6789/0 conn(0x7f9da81430c0 legacy=0x7f9da814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 1 monclient: found mon.noname-b 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:22:11.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da80b44c0 con 0x7f9da80a98f0 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9d9c0015b0 con 0x7f9da80a98f0 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9daef32640 5 monclient: authenticate success, global_id 4352 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9daef32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9da77fe640 10 monclient: dump: 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.019 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:11.020 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:11.020 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:11.020 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:11.020 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:11.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.016+0000 7f9daef32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d9c001710 con 0x7f9da80a98f0 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9da77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9da77fe640 1 -- 172.21.15.22:0/1956351092 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9d9c001b60 con 0x7f9da80a98f0 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9da77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9da77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9da77fe640 10 monclient: dump: 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:11.049 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:11.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 4 set_mon_vals no callback set 2024-03-25T18:22:11.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 10 monclient: shutdown 2024-03-25T18:22:11.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:11.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 1 -- 172.21.15.22:0/1956351092 >> v1:172.21.15.22:6790/0 conn(0x7f9da80a98f0 legacy=0x7f9da80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 1 -- 172.21.15.22:0/1956351092 shutdown_connections 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 1 -- 172.21.15.22:0/1956351092 >> 172.21.15.22:0/1956351092 conn(0x7f9da80b7ea0 msgr2=0x7f9da80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 1 -- 172.21.15.22:0/1956351092 shutdown_connections 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.048+0000 7f9daef32640 1 -- 172.21.15.22:0/1956351092 wait complete. 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.049+0000 7f9daef32640 10 monclient: build_initial_monmap 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.049+0000 7f9daef32640 10 monclient: monmap: 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:11.052 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.049+0000 7f9daef32640 1 librados: starting msgr at 2024-03-25T18:22:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.049+0000 7f9daef32640 1 librados: starting objecter 2024-03-25T18:22:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.050+0000 7f9daef32640 1 Processor -- start 2024-03-25T18:22:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.050+0000 7f9daef32640 1 -- start start 2024-03-25T18:22:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.050+0000 7f9daef32640 1 librados: setting wanted keys 2024-03-25T18:22:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.050+0000 7f9daef32640 1 librados: calling monclient init 2024-03-25T18:22:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.050+0000 7f9daef32640 10 monclient: init 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 10 monclient(hunting): picked mon.noname-a con 0x7f9da80a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 10 monclient(hunting): picked mon.noname-c con 0x7f9da813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 10 monclient(hunting): picked mon.noname-b con 0x7f9da81430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da81d78a0 con 0x7f9da80a98f0 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da81d51f0 con 0x7f9da81430c0 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9da81d8440 con 0x7f9da813f710 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.051+0000 7f9daef32640 10 monclient(hunting): authenticate will time out at 7330.275145s 2024-03-25T18:22:11.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.058+0000 7f9daccce640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f9da80a98f0 0x7f9da81d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:53676/0 (socket says 172.21.15.22:53676) 2024-03-25T18:22:11.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.058+0000 7f9daccce640 1 -- 172.21.15.22:0/3572625005 learned_addr learned my addr 172.21.15.22:0/3572625005 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:11.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.060+0000 7f9da57fa640 10 client.?.objecter ms_handle_connect 0x7f9da80a98f0 2024-03-25T18:22:11.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.060+0000 7f9da57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:11.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.063+0000 7f9da57fa640 10 client.?.objecter ms_handle_connect 0x7f9da81430c0 2024-03-25T18:22:11.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.063+0000 7f9da57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.070+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069310588 0 0) 0x7f9da81d78a0 con 0x7f9da80a98f0 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.071+0000 7f9da57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.071+0000 7f9da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.071+0000 7f9da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.071+0000 7f9da57fa640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.071+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d8c004eb0 con 0x7f9da80a98f0 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.071+0000 7f9da57fa640 10 client.?.objecter ms_handle_connect 0x7f9da813f710 2024-03-25T18:22:11.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.071+0000 7f9da57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:11.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.072+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157531346 0 0) 0x7f9da81d51f0 con 0x7f9da81430c0 2024-03-25T18:22:11.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.072+0000 7f9da57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.072+0000 7f9da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.072+0000 7f9da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.072+0000 7f9da57fa640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:22:11.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.072+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da81d78a0 con 0x7f9da81430c0 2024-03-25T18:22:11.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.077+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396795889 0 0) 0x7f9da81d8440 con 0x7f9da813f710 2024-03-25T18:22:11.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.077+0000 7f9da57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.077+0000 7f9da57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.077+0000 7f9da57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.077+0000 7f9da57fa640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:22:11.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.077+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9da81d51f0 con 0x7f9da813f710 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.079+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 329843361 0 0) 0x7f9d8c004eb0 con 0x7f9da80a98f0 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.079+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9da81d8440 con 0x7f9da80a98f0 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.079+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9d9c001b20 con 0x7f9da80a98f0 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.079+0000 7f9da57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.079+0000 7f9da57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.079+0000 7f9da57fa640 10 monclient(hunting): dump: 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.081 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:11.082 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:11.082 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:11.082 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:11.082 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:11.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.079+0000 7f9da57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:11.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.089+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209207236 0 0) 0x7f9da81d78a0 con 0x7f9da81430c0 2024-03-25T18:22:11.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.089+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d8c004eb0 con 0x7f9da81430c0 2024-03-25T18:22:11.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.089+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9da0001ab0 con 0x7f9da81430c0 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.089+0000 7f9da57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.089+0000 7f9da57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.089+0000 7f9da57fa640 10 monclient(hunting): dump: 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:11.091 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:11.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4187660429 0 0) 0x7f9da81d8440 con 0x7f9da80a98f0 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 >> v1:172.21.15.22:6790/0 conn(0x7f9da813f710 legacy=0x7f9da81385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 >> v1:172.21.15.33:6789/0 conn(0x7f9da81430c0 legacy=0x7f9da81e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 1 monclient: found mon.a 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da8138cb0 con 0x7f9da80a98f0 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9da57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 10 monclient: _renew_subs 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da8137890 con 0x7f9da80a98f0 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 10 monclient: _renew_subs 2024-03-25T18:22:11.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:11.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da81e3550 con 0x7f9da80a98f0 2024-03-25T18:22:11.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.095+0000 7f9daef32640 1 librados: init done 2024-03-25T18:22:11.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.096+0000 7f9daef32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:11.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.096+0000 7f9daef32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:11.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.096+0000 7f9daef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:11.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.096+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da81d7600 con 0x7f9da80a98f0 2024-03-25T18:22:11.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.399+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d9c0038a0 con 0x7f9da80a98f0 2024-03-25T18:22:11.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.399+0000 7f9da57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.399+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9d9c003c30 con 0x7f9da80a98f0 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.399+0000 7f9da57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.399+0000 7f9da57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.399+0000 7f9da57fa640 10 monclient: dump: 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:11.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.399+0000 7f9da6ffd640 4 set_mon_vals no callback set 2024-03-25T18:22:11.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.400+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f9d9c003e50 con 0x7f9da80a98f0 2024-03-25T18:22:11.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.400+0000 7f9da7fff640 10 monclient: get_auth_request con 0x7f9d8c06df70 auth_method 0 2024-03-25T18:22:11.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.400+0000 7f9da57fa640 10 client.4353.objecter ms_handle_connect 0x7f9d8c06df70 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f9d9c054890 con 0x7f9da80a98f0 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 10 client.4353.objecter ms_dispatch 0x7f9da8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:11.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T18:22:11.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9d9c0503a0 con 0x7f9da80a98f0 2024-03-25T18:22:11.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:11.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.408+0000 7f9da57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:11.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.618+0000 7f9daef32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:11.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.618+0000 7f9daef32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:11.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.618+0000 7f9daef32640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:11.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.618+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9da8074cc0 con 0x7f9da80a98f0 2024-03-25T18:22:11.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.641+0000 7f9da57fa640 1 -- 172.21.15.22:0/3572625005 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f9d9c0522d0 con 0x7f9da80a98f0 2024-03-25T18:22:11.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.641+0000 7f9da57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:11.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.641+0000 7f9da57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:11.643 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:11.643 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:11.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 10 librados: watch_flush enter 2024-03-25T18:22:11.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 10 librados: watch_flush exit 2024-03-25T18:22:11.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T18:22:11.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 10 client.4353.objecter successfully canceled tick 2024-03-25T18:22:11.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 >> v1:172.21.15.33:6800/215350958 conn(0x7f9d8c06df70 legacy=0x7f9d8c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 10 monclient: shutdown 2024-03-25T18:22:11.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:11.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.643+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 >> v1:172.21.15.22:6789/0 conn(0x7f9da80a98f0 legacy=0x7f9da81d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.644+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 shutdown_connections 2024-03-25T18:22:11.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.644+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 >> 172.21.15.22:0/3572625005 conn(0x7f9da80b7ea0 msgr2=0x7f9da8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:11.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.644+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 shutdown_connections 2024-03-25T18:22:11.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.644+0000 7f9daef32640 1 -- 172.21.15.22:0/3572625005 wait complete. 2024-03-25T18:22:11.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:11.644+0000 7f9daef32640 1 librados: shutdown 2024-03-25T18:22:11.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:12.350 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:15.012 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.405 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:15.068 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.443 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:15.105 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.460 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:15.078 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.464 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:15.126 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.472 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:15.090 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.477 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:15.139 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.494 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:15.113 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.511 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:15.173 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.461 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:18.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.579+0000 7f4f50ae6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.579+0000 7f4f50ae6640 10 monclient: build_initial_monmap 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.579+0000 7f4f50ae6640 10 monclient: monmap: 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:18.580986+0000 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:18.580986+0000 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:18.581 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:18.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.580+0000 7f4f50ae6640 1 Processor -- start 2024-03-25T18:22:18.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.580+0000 7f4f50ae6640 1 -- start start 2024-03-25T18:22:18.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.580+0000 7f4f50ae6640 10 monclient: init 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f4c0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f4c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c142520 con 0x7f4f4c1430c0 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0b28a0 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0b2a30 con 0x7f4f4c13f710 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:18.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.581+0000 7f4f50ae6640 10 monclient(hunting): authenticate will time out at 7337.805334s 2024-03-25T18:22:18.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.585+0000 7f4f49d9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4f4c13f710 0x7f4f4c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35026/0 (socket says 172.21.15.22:35026) 2024-03-25T18:22:18.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.585+0000 7f4f49d9b640 1 -- 172.21.15.22:0/2719135612 learned_addr learned my addr 172.21.15.22:0/2719135612 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:18.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.599+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225338209 0 0) 0x7f4f4c0b2a30 con 0x7f4f4c13f710 2024-03-25T18:22:18.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.599+0000 7f4f4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.599+0000 7f4f4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.599+0000 7f4f4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.599+0000 7f4f4959a640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:22:18.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.599+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f34003520 con 0x7f4f4c13f710 2024-03-25T18:22:18.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.609+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166437084 0 0) 0x7f4f4c0b28a0 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.609+0000 7f4f4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.609+0000 7f4f4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.609+0000 7f4f4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.609+0000 7f4f4959a640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:22:18.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.609+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f4c0b2a30 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.620+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037607893 0 0) 0x7f4f4c142520 con 0x7f4f4c1430c0 2024-03-25T18:22:18.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.620+0000 7f4f4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.620+0000 7f4f4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.620+0000 7f4f4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.620+0000 7f4f4959a640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:22:18.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.620+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f4c0b28a0 con 0x7f4f4c1430c0 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3344963748 0 0) 0x7f4f34003520 con 0x7f4f4c13f710 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 >> v1:172.21.15.33:6789/0 conn(0x7f4f4c0a98f0 legacy=0x7f4f4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 >> v1:172.21.15.22:6789/0 conn(0x7f4f4c1430c0 legacy=0x7f4f4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 10 monclient: _finish_hunting 0 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 1 monclient: found mon.noname-b 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0b44c0 con 0x7f4f4c13f710 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 10 monclient: _finish_auth 0 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 10 monclient: _check_auth_tickets 2024-03-25T18:22:18.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f40001690 con 0x7f4f4c13f710 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f4959a640 10 monclient: dump: 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f50ae6640 5 monclient: authenticate success, global_id 4358 2024-03-25T18:22:18.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.623+0000 7f4f50ae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:18.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f400017f0 con 0x7f4f4c13f710 2024-03-25T18:22:18.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:18.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f4959a640 1 -- 172.21.15.22:0/2719135612 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f40001c00 con 0x7f4f4c13f710 2024-03-25T18:22:18.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f50ae6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:18.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f50ae6640 4 set_mon_vals no callback set 2024-03-25T18:22:18.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f4959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f4959a640 10 monclient: dump: 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:18.639 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:18.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f50ae6640 10 monclient: shutdown 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f50ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.637+0000 7f4f50ae6640 1 -- 172.21.15.22:0/2719135612 >> v1:172.21.15.22:6790/0 conn(0x7f4f4c13f710 legacy=0x7f4f4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.638+0000 7f4f50ae6640 1 -- 172.21.15.22:0/2719135612 shutdown_connections 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.638+0000 7f4f50ae6640 1 -- 172.21.15.22:0/2719135612 >> 172.21.15.22:0/2719135612 conn(0x7f4f4c0b7ea0 msgr2=0x7f4f4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.638+0000 7f4f50ae6640 1 -- 172.21.15.22:0/2719135612 shutdown_connections 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.638+0000 7f4f50ae6640 1 -- 172.21.15.22:0/2719135612 wait complete. 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 10 monclient: build_initial_monmap 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 10 monclient: monmap: 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:18.641 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 1 librados: starting msgr at 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 1 librados: starting objecter 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 1 Processor -- start 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 1 -- start start 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 1 librados: setting wanted keys 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 1 librados: calling monclient init 2024-03-25T18:22:18.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.639+0000 7f4f50ae6640 10 monclient: init 2024-03-25T18:22:18.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:18.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:18.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f4c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:18.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:18.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f4c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:18.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c0aba00 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c1d7490 con 0x7f4f4c13f710 2024-03-25T18:22:18.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4c1d8360 con 0x7f4f4c1430c0 2024-03-25T18:22:18.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:18.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.640+0000 7f4f50ae6640 10 monclient(hunting): authenticate will time out at 7337.864241s 2024-03-25T18:22:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.649+0000 7f4f4ad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4f4c1430c0 0x7f4f4c133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35068/0 (socket says 172.21.15.22:35068) 2024-03-25T18:22:18.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.649+0000 7f4f4ad9d640 1 -- 172.21.15.22:0/1493675234 learned_addr learned my addr 172.21.15.22:0/1493675234 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:18.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.652+0000 7f4f3affd640 10 client.?.objecter ms_handle_connect 0x7f4f4c13f710 2024-03-25T18:22:18.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.652+0000 7f4f3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:18.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.652+0000 7f4f3affd640 10 client.?.objecter ms_handle_connect 0x7f4f4c1430c0 2024-03-25T18:22:18.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.652+0000 7f4f3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:18.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.655+0000 7f4f3affd640 10 client.?.objecter ms_handle_connect 0x7f4f4c0a98f0 2024-03-25T18:22:18.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.655+0000 7f4f3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:18.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712572063 0 0) 0x7f4f4c0aba00 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f2c004eb0 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481273992 0 0) 0x7f4f4c1d8360 con 0x7f4f4c1430c0 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.664+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f4c0aba00 con 0x7f4f4c1430c0 2024-03-25T18:22:18.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.665+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474309724 0 0) 0x7f4f4c1d7490 con 0x7f4f4c13f710 2024-03-25T18:22:18.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.665+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.665+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.665+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.665+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:22:18.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.665+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f4c1d8360 con 0x7f4f4c13f710 2024-03-25T18:22:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.674+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3369984066 0 0) 0x7f4f4c1d8360 con 0x7f4f4c13f710 2024-03-25T18:22:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.674+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f4c1d7490 con 0x7f4f4c13f710 2024-03-25T18:22:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.674+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f40001b00 con 0x7f4f4c13f710 2024-03-25T18:22:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.674+0000 7f4f3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.674+0000 7f4f3affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.674+0000 7f4f3affd640 10 monclient(hunting): dump: 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:18.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.674+0000 7f4f3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:18.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.681+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3262665392 0 0) 0x7f4f2c004eb0 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.681+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f4c1d8360 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.681+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f3c001970 con 0x7f4f4c0a98f0 2024-03-25T18:22:18.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.681+0000 7f4f3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.681+0000 7f4f3affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.681+0000 7f4f3affd640 10 monclient(hunting): dump: 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:18.683 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:18.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1590593645 0 0) 0x7f4f4c1d7490 con 0x7f4f4c13f710 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 >> v1:172.21.15.22:6790/0 conn(0x7f4f4c1430c0 legacy=0x7f4f4c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 >> v1:172.21.15.22:6789/0 conn(0x7f4f4c0a98f0 legacy=0x7f4f4c1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 1 monclient: found mon.b 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c0abc30 con 0x7f4f4c13f710 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 10 monclient: _finish_auth 0 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 10 monclient: _renew_subs 2024-03-25T18:22:18.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c137940 con 0x7f4f4c13f710 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 10 monclient: _renew_subs 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c135220 con 0x7f4f4c13f710 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.686+0000 7f4f50ae6640 1 librados: init done 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.687+0000 7f4f50ae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:18.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.687+0000 7f4f50ae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:18.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.687+0000 7f4f50ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:18.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.687+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c0627b0 con 0x7f4f4c13f710 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.997+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f40001c60 con 0x7f4f4c13f710 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.997+0000 7f4f3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.997+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f40004c30 con 0x7f4f4c13f710 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.997+0000 7f4f3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.997+0000 7f4f3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.997+0000 7f4f3affd640 10 monclient: dump: 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:18.999 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:19.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.997+0000 7f4f48d99640 4 set_mon_vals no callback set 2024-03-25T18:22:19.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.998+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f4f4001ac00 con 0x7f4f4c13f710 2024-03-25T18:22:19.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.999+0000 7f4f4a59c640 10 monclient: get_auth_request con 0x7f4f2c06ded0 auth_method 0 2024-03-25T18:22:19.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:18.999+0000 7f4f3affd640 10 client.4360.objecter ms_handle_connect 0x7f4f2c06ded0 2024-03-25T18:22:19.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.003+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f4f40054870 con 0x7f4f4c13f710 2024-03-25T18:22:19.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.003+0000 7f4f3affd640 10 client.4360.objecter ms_dispatch 0x7f4f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:19.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.003+0000 7f4f3affd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:19.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.003+0000 7f4f3affd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:19.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.003+0000 7f4f3affd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:22:19.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.004+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f40050380 con 0x7f4f4c13f710 2024-03-25T18:22:19.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.004+0000 7f4f3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:19.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.004+0000 7f4f3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:19.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.265+0000 7f4f50ae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.265+0000 7f4f50ae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.265+0000 7f4f50ae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:19.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.265+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f4c1d7830 con 0x7f4f4c13f710 2024-03-25T18:22:19.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.296+0000 7f4f3affd640 1 -- 172.21.15.22:0/1493675234 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f4f400522b0 con 0x7f4f4c13f710 2024-03-25T18:22:19.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.296+0000 7f4f3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.296+0000 7f4f3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:19.297 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:19.297 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:19.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 10 librados: watch_flush enter 2024-03-25T18:22:19.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 10 librados: watch_flush exit 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 >> v1:172.21.15.33:6800/215350958 conn(0x7f4f2c06ded0 legacy=0x7f4f2c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 10 monclient: shutdown 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 >> v1:172.21.15.33:6789/0 conn(0x7f4f4c13f710 legacy=0x7f4f4c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 shutdown_connections 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 >> 172.21.15.22:0/1493675234 conn(0x7f4f4c0b7ea0 msgr2=0x7f4f4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:19.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.298+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 shutdown_connections 2024-03-25T18:22:19.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.299+0000 7f4f50ae6640 1 -- 172.21.15.22:0/1493675234 wait complete. 2024-03-25T18:22:19.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:19.299+0000 7f4f50ae6640 1 librados: shutdown 2024-03-25T18:22:19.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:26.119 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.228+0000 7f6ff2a55640 10 monclient: get_monmap_and_config 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.228+0000 7f6ff2a55640 10 monclient: build_initial_monmap 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.228+0000 7f6ff2a55640 10 monclient: monmap: 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:26.230077+0000 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:26.230077+0000 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:26.230 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:26.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.229+0000 7f6ff2a55640 1 Processor -- start 2024-03-25T18:22:26.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.229+0000 7f6ff2a55640 1 -- start start 2024-03-25T18:22:26.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.229+0000 7f6ff2a55640 10 monclient: init 2024-03-25T18:22:26.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 10 monclient(hunting): picked mon.noname-a con 0x7f6fec0b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 10 monclient(hunting): picked mon.noname-b con 0x7f6fec0b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 10 monclient(hunting): picked mon.noname-c con 0x7f6fec14be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0af940 con 0x7f6fec0b4d60 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0b0a70 con 0x7f6fec14be90 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0b0d50 con 0x7f6fec0b1730 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:26.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.230+0000 7f6ff2a55640 10 monclient(hunting): authenticate will time out at 7345.454135s 2024-03-25T18:22:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.238+0000 7f6ff0ff2640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f6fec14be90 0x7f6fec14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:49458/0 (socket says 172.21.15.22:49458) 2024-03-25T18:22:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.238+0000 7f6ff0ff2640 1 -- 172.21.15.22:0/4269513663 learned_addr learned my addr 172.21.15.22:0/4269513663 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.256+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798026633 0 0) 0x7f6fec0b0a70 con 0x7f6fec14be90 2024-03-25T18:22:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.256+0000 7f6feaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.256+0000 7f6feaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.256+0000 7f6feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.256+0000 7f6feaffd640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:22:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.256+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4003520 con 0x7f6fec14be90 2024-03-25T18:22:26.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.269+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936223390 0 0) 0x7f6fec0af940 con 0x7f6fec0b4d60 2024-03-25T18:22:26.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.269+0000 7f6feaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.269+0000 7f6feaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.269+0000 7f6feaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.269+0000 7f6feaffd640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:22:26.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.269+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fec0b0a70 con 0x7f6fec0b4d60 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742678128 0 0) 0x7f6fd4003520 con 0x7f6fec14be90 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 >> v1:172.21.15.22:6790/0 conn(0x7f6fec0b1730 legacy=0x7f6fec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 >> v1:172.21.15.22:6789/0 conn(0x7f6fec0b4d60 legacy=0x7f6fec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 1 monclient: found mon.noname-c 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0b1030 con 0x7f6fec14be90 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:26.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fe4001a10 con 0x7f6fec14be90 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6feaffd640 10 monclient: dump: 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6ff2a55640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:22:26.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.274+0000 7f6ff2a55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fe4001b70 con 0x7f6fec14be90 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6feaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6feaffd640 1 -- 172.21.15.22:0/4269513663 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fe4001f80 con 0x7f6fec14be90 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6ff2a55640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6ff2a55640 4 set_mon_vals no callback set 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6feaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6feaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6feaffd640 10 monclient: dump: 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:26.289 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.290 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.290 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.290 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:26.290 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:26.290 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:26.290 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:26.290 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6ff2a55640 10 monclient: shutdown 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6ff2a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6ff2a55640 1 -- 172.21.15.22:0/4269513663 >> v1:172.21.15.33:6789/0 conn(0x7f6fec14be90 legacy=0x7f6fec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6ff2a55640 1 -- 172.21.15.22:0/4269513663 shutdown_connections 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.288+0000 7f6ff2a55640 1 -- 172.21.15.22:0/4269513663 >> 172.21.15.22:0/4269513663 conn(0x7f6fec0617a0 msgr2=0x7f6fec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.289+0000 7f6ff2a55640 1 -- 172.21.15.22:0/4269513663 shutdown_connections 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.289+0000 7f6ff2a55640 1 -- 172.21.15.22:0/4269513663 wait complete. 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.289+0000 7f6ff2a55640 10 monclient: build_initial_monmap 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.289+0000 7f6ff2a55640 10 monclient: monmap: 2024-03-25T18:22:26.291 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.290+0000 7f6ff2a55640 1 librados: starting msgr at 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.290+0000 7f6ff2a55640 1 librados: starting objecter 2024-03-25T18:22:26.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.290+0000 7f6ff2a55640 1 Processor -- start 2024-03-25T18:22:26.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.290+0000 7f6ff2a55640 1 -- start start 2024-03-25T18:22:26.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.290+0000 7f6ff2a55640 1 librados: setting wanted keys 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.290+0000 7f6ff2a55640 1 librados: calling monclient init 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.290+0000 7f6ff2a55640 10 monclient: init 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.291+0000 7f6ff2a55640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.291+0000 7f6ff2a55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.291+0000 7f6ff2a55640 10 monclient(hunting): picked mon.noname-c con 0x7f6fec0b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.291+0000 7f6ff2a55640 10 monclient(hunting): picked mon.noname-b con 0x7f6fec0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.291+0000 7f6ff2a55640 10 monclient(hunting): picked mon.noname-a con 0x7f6fec14be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.291+0000 7f6ff2a55640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec137a30 con 0x7f6fec14be90 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.292+0000 7f6ff2a55640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0a9470 con 0x7f6fec0b4d60 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.292+0000 7f6ff2a55640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fec0aa5f0 con 0x7f6fec0b1730 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.292+0000 7f6ff2a55640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:26.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.292+0000 7f6ff2a55640 10 monclient(hunting): authenticate will time out at 7345.515723s 2024-03-25T18:22:26.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.293+0000 7f6febfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6fec0b1730 0x7f6fec138e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41004/0 (socket says 172.21.15.22:41004) 2024-03-25T18:22:26.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.294+0000 7f6febfff640 1 -- 172.21.15.22:0/3035639049 learned_addr learned my addr 172.21.15.22:0/3035639049 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:26.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.297+0000 7f6fe8ff9640 10 client.?.objecter ms_handle_connect 0x7f6fec0b1730 2024-03-25T18:22:26.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.297+0000 7f6fe8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.309+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814940111 0 0) 0x7f6fec0aa5f0 con 0x7f6fec0b1730 2024-03-25T18:22:26.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.309+0000 7f6fe8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.309+0000 7f6fe8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.309+0000 7f6fe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.309+0000 7f6fe8ff9640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:22:26.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.309+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd8004eb0 con 0x7f6fec0b1730 2024-03-25T18:22:26.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.313+0000 7f6fe8ff9640 10 client.?.objecter ms_handle_connect 0x7f6fec0b4d60 2024-03-25T18:22:26.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.313+0000 7f6fe8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.317+0000 7f6fe8ff9640 10 client.?.objecter ms_handle_connect 0x7f6fec14be90 2024-03-25T18:22:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.317+0000 7f6fe8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.322+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638628115 0 0) 0x7f6fd8004eb0 con 0x7f6fec0b1730 2024-03-25T18:22:26.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.322+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fec0aa5f0 con 0x7f6fec0b1730 2024-03-25T18:22:26.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.322+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fdc001a10 con 0x7f6fec0b1730 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.322+0000 7f6fe8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.322+0000 7f6fe8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.322+0000 7f6fe8ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:26.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.322+0000 7f6fe8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:26.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.333+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497360944 0 0) 0x7f6fec137a30 con 0x7f6fec14be90 2024-03-25T18:22:26.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.333+0000 7f6fe8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.333+0000 7f6fe8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.333+0000 7f6fe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.333+0000 7f6fe8ff9640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:22:26.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.333+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd8004eb0 con 0x7f6fec14be90 2024-03-25T18:22:26.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.343+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863262716 0 0) 0x7f6fec0a9470 con 0x7f6fec0b4d60 2024-03-25T18:22:26.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.343+0000 7f6fe8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.343+0000 7f6fe8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.343+0000 7f6fe8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.343+0000 7f6fe8ff9640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:22:26.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.343+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fec137a30 con 0x7f6fec0b4d60 2024-03-25T18:22:26.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2949145598 0 0) 0x7f6fec0aa5f0 con 0x7f6fec0b1730 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 >> v1:172.21.15.33:6789/0 conn(0x7f6fec0b4d60 legacy=0x7f6fec136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 >> v1:172.21.15.22:6789/0 conn(0x7f6fec14be90 legacy=0x7f6fec137320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 1 monclient: found mon.c 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fec0aa8d0 con 0x7f6fec0b1730 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6fe8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 5 monclient: authenticate success, global_id 4367 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 10 monclient: _renew_subs 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec0aabb0 con 0x7f6fec0b1730 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:26.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 10 monclient: _renew_subs 2024-03-25T18:22:26.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:22:26.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec0aa180 con 0x7f6fec0b1730 2024-03-25T18:22:26.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.345+0000 7f6ff2a55640 1 librados: init done 2024-03-25T18:22:26.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.347+0000 7f6ff2a55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.347+0000 7f6ff2a55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.347+0000 7f6ff2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:22:26.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.347+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec0627b0 con 0x7f6fec0b1730 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.354+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fdc001c90 con 0x7f6fec0b1730 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.354+0000 7f6fe8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.354+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fdc003a60 con 0x7f6fec0b1730 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.354+0000 7f6fe8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.354+0000 7f6fe8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.354+0000 7f6fe8ff9640 10 monclient: dump: 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:26.355 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:26.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.354+0000 7f6fea7fc640 4 set_mon_vals no callback set 2024-03-25T18:22:26.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.669+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f6fdc003bc0 con 0x7f6fec0b1730 2024-03-25T18:22:26.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.670+0000 7f6feb7fe640 10 monclient: get_auth_request con 0x7f6fd806de80 auth_method 0 2024-03-25T18:22:26.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.670+0000 7f6fe8ff9640 10 client.4367.objecter ms_handle_connect 0x7f6fd806de80 2024-03-25T18:22:26.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f6fdc054890 con 0x7f6fec0b1730 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 10 client.4367.objecter ms_dispatch 0x7f6fec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fdc0503a0 con 0x7f6fec0b1730 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.676+0000 7f6fe8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:26.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.926+0000 7f6ff2a55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:26.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.926+0000 7f6ff2a55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:26.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.926+0000 7f6ff2a55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:22:26.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.926+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fec069f40 con 0x7f6fec0b1730 2024-03-25T18:22:26.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.952+0000 7f6fe8ff9640 1 -- 172.21.15.22:0/3035639049 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f6fdc0522d0 con 0x7f6fec0b1730 2024-03-25T18:22:26.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.952+0000 7f6fe8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:26.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.952+0000 7f6fe8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:26.953 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:26.954 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 10 librados: watch_flush enter 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 10 librados: watch_flush exit 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 10 client.4367.objecter successfully canceled tick 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 >> v1:172.21.15.33:6800/215350958 conn(0x7f6fd806de80 legacy=0x7f6fd8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 10 monclient: shutdown 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 >> v1:172.21.15.22:6790/0 conn(0x7f6fec0b1730 legacy=0x7f6fec138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 shutdown_connections 2024-03-25T18:22:26.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 >> 172.21.15.22:0/3035639049 conn(0x7f6fec0617a0 msgr2=0x7f6fec145100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:26.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 shutdown_connections 2024-03-25T18:22:26.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.954+0000 7f6ff2a55640 1 -- 172.21.15.22:0/3035639049 wait complete. 2024-03-25T18:22:26.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:26.955+0000 7f6ff2a55640 1 librados: shutdown 2024-03-25T18:22:26.968 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:32.422 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:35.084 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.482 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:35.144 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.513 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:35.175 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.549 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:35.166 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.549 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:35.211 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.555 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:35.217 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.558 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:35.176 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.574 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:35.236 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:32.590 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:35.208 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:33.776 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:33.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.891+0000 7f9f287c8640 10 monclient: get_monmap_and_config 2024-03-25T18:22:33.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.891+0000 7f9f287c8640 10 monclient: build_initial_monmap 2024-03-25T18:22:33.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.892+0000 7f9f287c8640 10 monclient: monmap: 2024-03-25T18:22:33.893 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:33.893293+0000 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:33.893293+0000 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:33.894 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:33.895 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.892+0000 7f9f287c8640 1 Processor -- start 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.892+0000 7f9f287c8640 1 -- start start 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.892+0000 7f9f287c8640 10 monclient: init 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.893+0000 7f9f287c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.893+0000 7f9f287c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f2013fc60 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f20070570 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:33.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f200727a0 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2013c9f0 con 0x7f9f20070570 2024-03-25T18:22:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f2006fac0 con 0x7f9f200727a0 2024-03-25T18:22:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200715d0 con 0x7f9f2013fc60 2024-03-25T18:22:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:33.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.894+0000 7f9f287c8640 10 monclient(hunting): authenticate will time out at 7353.117806s 2024-03-25T18:22:33.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.906+0000 7f9f26564640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f9f2013fc60 0x7f9f2006f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:33306/0 (socket says 172.21.15.22:33306) 2024-03-25T18:22:33.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.906+0000 7f9f26564640 1 -- 172.21.15.22:0/4123305934 learned_addr learned my addr 172.21.15.22:0/4123305934 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:33.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.929+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859950603 0 0) 0x7f9f2006fac0 con 0x7f9f200727a0 2024-03-25T18:22:33.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.929+0000 7f9f25562640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.929+0000 7f9f25562640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.929+0000 7f9f25562640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.929+0000 7f9f25562640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:22:33.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.929+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c003520 con 0x7f9f200727a0 2024-03-25T18:22:33.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.930+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119016320 0 0) 0x7f9f200715d0 con 0x7f9f2013fc60 2024-03-25T18:22:33.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.930+0000 7f9f25562640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.930+0000 7f9f25562640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.930+0000 7f9f25562640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.930+0000 7f9f25562640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:22:33.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.930+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f2006fac0 con 0x7f9f2013fc60 2024-03-25T18:22:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.942+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044158124 0 0) 0x7f9f2013c9f0 con 0x7f9f20070570 2024-03-25T18:22:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.942+0000 7f9f25562640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.942+0000 7f9f25562640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.942+0000 7f9f25562640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.942+0000 7f9f25562640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:22:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.942+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f200715d0 con 0x7f9f20070570 2024-03-25T18:22:33.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1426755628 0 0) 0x7f9f2006fac0 con 0x7f9f2013fc60 2024-03-25T18:22:33.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 >> v1:172.21.15.33:6789/0 conn(0x7f9f200727a0 legacy=0x7f9f2014a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 >> v1:172.21.15.22:6789/0 conn(0x7f9f20070570 legacy=0x7f9f20072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 10 monclient: _finish_hunting 0 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 1 monclient: found mon.noname-b 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f200718b0 con 0x7f9f2013fc60 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 10 monclient: _finish_auth 0 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 10 monclient: _check_auth_tickets 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f1c001670 con 0x7f9f2013fc60 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f25562640 10 monclient: dump: 2024-03-25T18:22:33.945 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f287c8640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:22:33.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.943+0000 7f9f287c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f1c0017d0 con 0x7f9f2013fc60 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f25562640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f25562640 1 -- 172.21.15.22:0/4123305934 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f1c001be0 con 0x7f9f2013fc60 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f287c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f287c8640 4 set_mon_vals no callback set 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f25562640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f25562640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.973+0000 7f9f25562640 10 monclient: dump: 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:33.975 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:33.976 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:33.976 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:33.976 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:33.976 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:33.976 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:33.976 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:33.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.974+0000 7f9f287c8640 10 monclient: shutdown 2024-03-25T18:22:33.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.974+0000 7f9f287c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:33.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.974+0000 7f9f287c8640 1 -- 172.21.15.22:0/4123305934 >> v1:172.21.15.22:6790/0 conn(0x7f9f2013fc60 legacy=0x7f9f2006f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.974+0000 7f9f287c8640 1 -- 172.21.15.22:0/4123305934 shutdown_connections 2024-03-25T18:22:33.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.974+0000 7f9f287c8640 1 -- 172.21.15.22:0/4123305934 >> 172.21.15.22:0/4123305934 conn(0x7f9f200b4a80 msgr2=0x7f9f200b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:33.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.974+0000 7f9f287c8640 1 -- 172.21.15.22:0/4123305934 shutdown_connections 2024-03-25T18:22:33.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.974+0000 7f9f287c8640 1 -- 172.21.15.22:0/4123305934 wait complete. 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.975+0000 7f9f287c8640 10 monclient: build_initial_monmap 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.975+0000 7f9f287c8640 10 monclient: monmap: 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.975+0000 7f9f287c8640 1 librados: starting msgr at 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.975+0000 7f9f287c8640 1 librados: starting objecter 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.975+0000 7f9f287c8640 1 Processor -- start 2024-03-25T18:22:33.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 1 -- start start 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 1 librados: setting wanted keys 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 1 librados: calling monclient init 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient: init 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f20070570 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f200727a0 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f2013fc60 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200ad3d0 con 0x7f9f2013fc60 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f20132610 con 0x7f9f20070570 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200ae0e0 con 0x7f9f200727a0 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.976+0000 7f9f287c8640 10 monclient(hunting): authenticate will time out at 7353.200540s 2024-03-25T18:22:33.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.982+0000 7f9f26d65640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f9f2013fc60 0x7f9f201e03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:40800/0 (socket says 172.21.15.22:40800) 2024-03-25T18:22:33.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.982+0000 7f9f26d65640 1 -- 172.21.15.22:0/1554606018 learned_addr learned my addr 172.21.15.22:0/1554606018 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:33.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.983+0000 7f9f16ffd640 10 client.?.objecter ms_handle_connect 0x7f9f2013fc60 2024-03-25T18:22:33.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.983+0000 7f9f16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:33.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.987+0000 7f9f16ffd640 10 client.?.objecter ms_handle_connect 0x7f9f20070570 2024-03-25T18:22:33.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.987+0000 7f9f16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:33.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.989+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2041570724 0 0) 0x7f9f200ad3d0 con 0x7f9f2013fc60 2024-03-25T18:22:33.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.989+0000 7f9f16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.989+0000 7f9f16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.989+0000 7f9f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.989+0000 7f9f16ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:22:33.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.989+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f04004eb0 con 0x7f9f2013fc60 2024-03-25T18:22:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.995+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107159610 0 0) 0x7f9f20132610 con 0x7f9f20070570 2024-03-25T18:22:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.995+0000 7f9f16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.995+0000 7f9f16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.995+0000 7f9f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.995+0000 7f9f16ffd640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:22:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.995+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f200ad3d0 con 0x7f9f20070570 2024-03-25T18:22:33.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.997+0000 7f9f16ffd640 10 client.?.objecter ms_handle_connect 0x7f9f200727a0 2024-03-25T18:22:33.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:33.997+0000 7f9f16ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:34.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.003+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108746795 0 0) 0x7f9f04004eb0 con 0x7f9f2013fc60 2024-03-25T18:22:34.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.003+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f20132610 con 0x7f9f2013fc60 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.003+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f18001c50 con 0x7f9f2013fc60 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.003+0000 7f9f16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.003+0000 7f9f16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.003+0000 7f9f16ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:34.005 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:34.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.003+0000 7f9f16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:34.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.004+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487098245 0 0) 0x7f9f200ae0e0 con 0x7f9f200727a0 2024-03-25T18:22:34.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.004+0000 7f9f16ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:34.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.004+0000 7f9f16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:34.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.004+0000 7f9f16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:34.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.004+0000 7f9f16ffd640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:22:34.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.004+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f04004eb0 con 0x7f9f200727a0 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.010+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3156449580 0 0) 0x7f9f200ad3d0 con 0x7f9f20070570 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.010+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f200ae0e0 con 0x7f9f20070570 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.010+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f1c001ba0 con 0x7f9f20070570 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.010+0000 7f9f16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.010+0000 7f9f16ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.010+0000 7f9f16ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:34.012 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:34.013 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:34.013 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:34.013 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2412295705 0 0) 0x7f9f20132610 con 0x7f9f2013fc60 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 >> v1:172.21.15.22:6790/0 conn(0x7f9f200727a0 legacy=0x7f9f200ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 >> v1:172.21.15.33:6789/0 conn(0x7f9f20070570 legacy=0x7f9f200abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 1 monclient: found mon.a 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f200ad600 con 0x7f9f2013fc60 2024-03-25T18:22:34.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 10 monclient: _renew_subs 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f200ad940 con 0x7f9f2013fc60 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 10 monclient: _renew_subs 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f200ade20 con 0x7f9f2013fc60 2024-03-25T18:22:34.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.017+0000 7f9f287c8640 1 librados: init done 2024-03-25T18:22:34.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.018+0000 7f9f287c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:34.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.018+0000 7f9f287c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:34.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.018+0000 7f9f287c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:34.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.018+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f200627b0 con 0x7f9f2013fc60 2024-03-25T18:22:34.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f18003990 con 0x7f9f2013fc60 2024-03-25T18:22:34.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:34.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9f18003da0 con 0x7f9f2013fc60 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f16ffd640 10 monclient: dump: 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:34.333 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:34.334 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f24d61640 4 set_mon_vals no callback set 2024-03-25T18:22:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.332+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f9f18003fc0 con 0x7f9f2013fc60 2024-03-25T18:22:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.333+0000 7f9f26564640 10 monclient: get_auth_request con 0x7f9f0406dfb0 auth_method 0 2024-03-25T18:22:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.333+0000 7f9f16ffd640 10 client.4371.objecter ms_handle_connect 0x7f9f0406dfb0 2024-03-25T18:22:34.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f9f18054ac0 con 0x7f9f2013fc60 2024-03-25T18:22:34.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 10 client.4371.objecter ms_dispatch 0x7f9f20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:34.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:34.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:34.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T18:22:34.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f180505d0 con 0x7f9f2013fc60 2024-03-25T18:22:34.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:34.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.340+0000 7f9f16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:34.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.585+0000 7f9f287c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:34.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.585+0000 7f9f287c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:34.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.585+0000 7f9f287c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:34.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.585+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f200ae290 con 0x7f9f2013fc60 2024-03-25T18:22:34.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.635+0000 7f9f16ffd640 1 -- 172.21.15.22:0/1554606018 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f9f18052500 con 0x7f9f2013fc60 2024-03-25T18:22:34.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.635+0000 7f9f16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:34.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.635+0000 7f9f16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:34.636 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:34.637 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 10 librados: watch_flush enter 2024-03-25T18:22:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 10 librados: watch_flush exit 2024-03-25T18:22:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T18:22:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 10 client.4371.objecter successfully canceled tick 2024-03-25T18:22:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 >> v1:172.21.15.33:6800/215350958 conn(0x7f9f0406dfb0 legacy=0x7f9f04070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 10 monclient: shutdown 2024-03-25T18:22:34.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:34.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 >> v1:172.21.15.22:6789/0 conn(0x7f9f2013fc60 legacy=0x7f9f201e03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 shutdown_connections 2024-03-25T18:22:34.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 >> 172.21.15.22:0/1554606018 conn(0x7f9f200b4a80 msgr2=0x7f9f20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:34.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 shutdown_connections 2024-03-25T18:22:34.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.638+0000 7f9f287c8640 1 -- 172.21.15.22:0/1554606018 wait complete. 2024-03-25T18:22:34.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:34.639+0000 7f9f287c8640 1 librados: shutdown 2024-03-25T18:22:34.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:41.458 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:41.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.573+0000 7f36a52d0640 10 monclient: get_monmap_and_config 2024-03-25T18:22:41.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.573+0000 7f36a52d0640 10 monclient: build_initial_monmap 2024-03-25T18:22:41.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.573+0000 7f36a52d0640 10 monclient: monmap: 2024-03-25T18:22:41.573 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:41.573431+0000 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:41.573431+0000 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:41.574 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:41.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.574+0000 7f36a52d0640 1 Processor -- start 2024-03-25T18:22:41.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.574+0000 7f36a52d0640 1 -- start start 2024-03-25T18:22:41.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.574+0000 7f36a52d0640 10 monclient: init 2024-03-25T18:22:41.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.574+0000 7f36a52d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:41.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:41.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 10 monclient(hunting): picked mon.noname-b con 0x7f36a00a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:41.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 10 monclient(hunting): picked mon.noname-c con 0x7f36a013f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:41.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 10 monclient(hunting): picked mon.noname-a con 0x7f36a01430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:41.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a0142520 con 0x7f36a01430c0 2024-03-25T18:22:41.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a00b28a0 con 0x7f36a013f710 2024-03-25T18:22:41.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a00b2a30 con 0x7f36a00a98f0 2024-03-25T18:22:41.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:41.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.575+0000 7f36a52d0640 10 monclient(hunting): authenticate will time out at 7360.797750s 2024-03-25T18:22:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.581+0000 7f369effd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f36a00a98f0 0x7f36a00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37842/0 (socket says 172.21.15.22:37842) 2024-03-25T18:22:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.581+0000 7f369effd640 1 -- 172.21.15.22:0/1309515746 learned_addr learned my addr 172.21.15.22:0/1309515746 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:41.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.605+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 537297332 0 0) 0x7f36a00b2a30 con 0x7f36a00a98f0 2024-03-25T18:22:41.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.605+0000 7f369dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.605+0000 7f369dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.605+0000 7f369dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.605+0000 7f369dffb640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:22:41.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.605+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3688003520 con 0x7f36a00a98f0 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.605+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212509352 0 0) 0x7f36a0142520 con 0x7f36a01430c0 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a00b2a30 con 0x7f36a01430c0 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853273158 0 0) 0x7f36a00b28a0 con 0x7f36a013f710 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.606+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0142520 con 0x7f36a013f710 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507487111 0 0) 0x7f36a0142520 con 0x7f36a013f710 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 >> v1:172.21.15.22:6790/0 conn(0x7f36a00a98f0 legacy=0x7f36a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 >> v1:172.21.15.22:6789/0 conn(0x7f36a01430c0 legacy=0x7f36a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 1 monclient: found mon.noname-c 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a00b44c0 con 0x7f36a013f710 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.618+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f36940015b0 con 0x7f36a013f710 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.619+0000 7f369dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.619+0000 7f369dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.619+0000 7f369dffb640 10 monclient: dump: 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:41.621 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:41.621 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.619+0000 7f36a52d0640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:22:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.619+0000 7f36a52d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3694001710 con 0x7f36a013f710 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f369dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f369dffb640 1 -- 172.21.15.22:0/1309515746 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3694001b60 con 0x7f36a013f710 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f369dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f369dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f369dffb640 10 monclient: dump: 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:41.626 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:41.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f36a52d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:41.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.624+0000 7f36a52d0640 4 set_mon_vals no callback set 2024-03-25T18:22:41.627 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.625+0000 7f36a52d0640 10 monclient: shutdown 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.625+0000 7f36a52d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.625+0000 7f36a52d0640 1 -- 172.21.15.22:0/1309515746 >> v1:172.21.15.33:6789/0 conn(0x7f36a013f710 legacy=0x7f36a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.625+0000 7f36a52d0640 1 -- 172.21.15.22:0/1309515746 shutdown_connections 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.625+0000 7f36a52d0640 1 -- 172.21.15.22:0/1309515746 >> 172.21.15.22:0/1309515746 conn(0x7f36a00b7ea0 msgr2=0x7f36a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.625+0000 7f36a52d0640 1 -- 172.21.15.22:0/1309515746 shutdown_connections 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.625+0000 7f36a52d0640 1 -- 172.21.15.22:0/1309515746 wait complete. 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.626+0000 7f36a52d0640 10 monclient: build_initial_monmap 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.626+0000 7f36a52d0640 10 monclient: monmap: 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:41.628 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.626+0000 7f36a52d0640 1 librados: starting msgr at 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.626+0000 7f36a52d0640 1 librados: starting objecter 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 1 Processor -- start 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 1 -- start start 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 1 librados: setting wanted keys 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 1 librados: calling monclient init 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient: init 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient(hunting): picked mon.noname-a con 0x7f36a00a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient(hunting): picked mon.noname-b con 0x7f36a013f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient(hunting): picked mon.noname-c con 0x7f36a01430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a01db7f0 con 0x7f36a00a98f0 2024-03-25T18:22:41.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a00abd70 con 0x7f36a013f710 2024-03-25T18:22:41.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a0138ca0 con 0x7f36a01430c0 2024-03-25T18:22:41.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:41.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.627+0000 7f36a52d0640 10 monclient(hunting): authenticate will time out at 7360.850341s 2024-03-25T18:22:41.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.641+0000 7f369effd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f36a00a98f0 0x7f36a0137c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:45094/0 (socket says 172.21.15.22:45094) 2024-03-25T18:22:41.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.641+0000 7f369effd640 1 -- 172.21.15.22:0/1427193259 learned_addr learned my addr 172.21.15.22:0/1427193259 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:41.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.643+0000 7f36877fe640 10 client.?.objecter ms_handle_connect 0x7f36a00a98f0 2024-03-25T18:22:41.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.643+0000 7f36877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:41.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.645+0000 7f36877fe640 10 client.?.objecter ms_handle_connect 0x7f36a013f710 2024-03-25T18:22:41.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.645+0000 7f36877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:41.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.654+0000 7f36877fe640 10 client.?.objecter ms_handle_connect 0x7f36a01430c0 2024-03-25T18:22:41.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.654+0000 7f36877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.656+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871639505 0 0) 0x7f36a00abd70 con 0x7f36a013f710 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.656+0000 7f36877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.656+0000 7f36877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.656+0000 7f36877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.656+0000 7f36877fe640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.656+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f368c004eb0 con 0x7f36a013f710 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.657+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619905167 0 0) 0x7f36a01db7f0 con 0x7f36a00a98f0 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.657+0000 7f36877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.657+0000 7f36877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.657+0000 7f36877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.657+0000 7f36877fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:22:41.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.657+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a00abd70 con 0x7f36a00a98f0 2024-03-25T18:22:41.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.667+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 613571305 0 0) 0x7f368c004eb0 con 0x7f36a013f710 2024-03-25T18:22:41.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.667+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36a01db7f0 con 0x7f36a013f710 2024-03-25T18:22:41.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.667+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3694001a20 con 0x7f36a013f710 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.667+0000 7f36877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.667+0000 7f36877fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.667+0000 7f36877fe640 10 monclient(hunting): dump: 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:41.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.667+0000 7f36877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:41.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.668+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271643753 0 0) 0x7f36a0138ca0 con 0x7f36a01430c0 2024-03-25T18:22:41.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.668+0000 7f36877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.668+0000 7f36877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.668+0000 7f36877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.668+0000 7f36877fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:22:41.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.668+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f368c004eb0 con 0x7f36a01430c0 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.672+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 672768042 0 0) 0x7f36a00abd70 con 0x7f36a00a98f0 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.672+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36a0138ca0 con 0x7f36a00a98f0 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.672+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f36900018d0 con 0x7f36a00a98f0 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.672+0000 7f36877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.672+0000 7f36877fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.672+0000 7f36877fe640 10 monclient(hunting): dump: 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:41.672 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.673 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:41.673 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.673 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:41.673 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:41.673 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:41.673 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:41.673 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:41.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3246685114 0 0) 0x7f36a0138ca0 con 0x7f36a00a98f0 2024-03-25T18:22:41.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 >> v1:172.21.15.22:6790/0 conn(0x7f36a01430c0 legacy=0x7f36a0138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 >> v1:172.21.15.33:6789/0 conn(0x7f36a013f710 legacy=0x7f36a01db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 1 monclient: found mon.a 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a0138e30 con 0x7f36a00a98f0 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 5 monclient: authenticate success, global_id 4377 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 10 monclient: _renew_subs 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a01e2c30 con 0x7f36a00a98f0 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 10 monclient: _renew_subs 2024-03-25T18:22:41.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:41.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a01e2fa0 con 0x7f36a00a98f0 2024-03-25T18:22:41.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.684+0000 7f36a52d0640 1 librados: init done 2024-03-25T18:22:41.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.685+0000 7f36a52d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:41.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.685+0000 7f36a52d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:41.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.685+0000 7f36a52d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:41.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:41.685+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a00b3ef0 con 0x7f36a00a98f0 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.017+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36900047e0 con 0x7f36a00a98f0 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.017+0000 7f36877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.017+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3690004af0 con 0x7f36a00a98f0 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.017+0000 7f36877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.017+0000 7f36877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.017+0000 7f36877fe640 10 monclient: dump: 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:42.017 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:42.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.017+0000 7f369d7fa640 4 set_mon_vals no callback set 2024-03-25T18:22:42.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.018+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f369001aac0 con 0x7f36a00a98f0 2024-03-25T18:22:42.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.018+0000 7f369e7fc640 10 monclient: get_auth_request con 0x7f368c06dec0 auth_method 0 2024-03-25T18:22:42.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.018+0000 7f36877fe640 10 client.4377.objecter ms_handle_connect 0x7f368c06dec0 2024-03-25T18:22:42.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f36900547d0 con 0x7f36a00a98f0 2024-03-25T18:22:42.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 10 client.4377.objecter ms_dispatch 0x7f36a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:42.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:42.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:42.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-25T18:22:42.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3690050110 con 0x7f36a00a98f0 2024-03-25T18:22:42.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:42.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.025+0000 7f36877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:42.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.242+0000 7f36a52d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.242+0000 7f36a52d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.242+0000 7f36a52d0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:42.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.242+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36a00627b0 con 0x7f36a00a98f0 2024-03-25T18:22:42.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.282+0000 7f36877fe640 1 -- 172.21.15.22:0/1427193259 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f3690052210 con 0x7f36a00a98f0 2024-03-25T18:22:42.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.282+0000 7f36877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.282+0000 7f36877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:42.282 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:42.282 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:42.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 10 librados: watch_flush enter 2024-03-25T18:22:42.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 10 librados: watch_flush exit 2024-03-25T18:22:42.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-25T18:22:42.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 10 client.4377.objecter successfully canceled tick 2024-03-25T18:22:42.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 >> v1:172.21.15.33:6800/215350958 conn(0x7f368c06dec0 legacy=0x7f368c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 10 monclient: shutdown 2024-03-25T18:22:42.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:42.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 >> v1:172.21.15.22:6789/0 conn(0x7f36a00a98f0 legacy=0x7f36a0137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 shutdown_connections 2024-03-25T18:22:42.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.284+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 >> 172.21.15.22:0/1427193259 conn(0x7f36a00b7ea0 msgr2=0x7f36a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:42.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.285+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 shutdown_connections 2024-03-25T18:22:42.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.285+0000 7f36a52d0640 1 -- 172.21.15.22:0/1427193259 wait complete. 2024-03-25T18:22:42.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:42.285+0000 7f36a52d0640 1 librados: shutdown 2024-03-25T18:22:42.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:49.104 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:49.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.219+0000 7f0ba004a640 10 monclient: get_monmap_and_config 2024-03-25T18:22:49.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.219+0000 7f0ba004a640 10 monclient: build_initial_monmap 2024-03-25T18:22:49.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.219+0000 7f0ba004a640 10 monclient: monmap: 2024-03-25T18:22:49.219 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:49.219 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:49.219 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:49.219407+0000 2024-03-25T18:22:49.219 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:49.219407+0000 2024-03-25T18:22:49.220 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:49.220 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:49.220 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:49.220 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:49.220 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:49.220 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.219+0000 7f0ba004a640 1 Processor -- start 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.219+0000 7f0ba004a640 1 -- start start 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.219+0000 7f0ba004a640 10 monclient: init 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 10 monclient(hunting): picked mon.noname-a con 0x7f0b980b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 10 monclient(hunting): picked mon.noname-b con 0x7f0b980b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 10 monclient(hunting): picked mon.noname-c con 0x7f0b9814be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:49.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980af940 con 0x7f0b980b4d60 2024-03-25T18:22:49.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980b0a70 con 0x7f0b9814be90 2024-03-25T18:22:49.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980b0d50 con 0x7f0b980b1730 2024-03-25T18:22:49.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:49.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.220+0000 7f0ba004a640 10 monclient(hunting): authenticate will time out at 7368.443450s 2024-03-25T18:22:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.226+0000 7f0b9dde6640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f0b980b4d60 0x7f0b9813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:45242/0 (socket says 172.21.15.22:45242) 2024-03-25T18:22:49.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.226+0000 7f0b9dde6640 1 -- 172.21.15.22:0/2584603202 learned_addr learned my addr 172.21.15.22:0/2584603202 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.237+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762152000 0 0) 0x7f0b980af940 con 0x7f0b980b4d60 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.237+0000 7f0b9cde4640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.237+0000 7f0b9cde4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.237+0000 7f0b9cde4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.237+0000 7f0b9cde4640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.237+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b84003520 con 0x7f0b980b4d60 2024-03-25T18:22:49.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.246+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079492062 0 0) 0x7f0b980b0d50 con 0x7f0b980b1730 2024-03-25T18:22:49.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.246+0000 7f0b9cde4640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.246+0000 7f0b9cde4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.246+0000 7f0b9cde4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.246+0000 7f0b9cde4640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:22:49.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.246+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b980af940 con 0x7f0b980b1730 2024-03-25T18:22:49.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.247+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117368506 0 0) 0x7f0b84003520 con 0x7f0b980b4d60 2024-03-25T18:22:49.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 >> v1:172.21.15.22:6790/0 conn(0x7f0b980b1730 legacy=0x7f0b980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 >> v1:172.21.15.33:6789/0 conn(0x7f0b9814be90 legacy=0x7f0b9814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 10 monclient: _finish_hunting 0 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 1 monclient: found mon.noname-a 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b980b1030 con 0x7f0b980b4d60 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 10 monclient: _finish_auth 0 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 10 monclient: _check_auth_tickets 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0b94001690 con 0x7f0b980b4d60 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0b9cde4640 10 monclient: dump: 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:49.249 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0ba004a640 5 monclient: authenticate success, global_id 4380 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.248+0000 7f0ba004a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:49.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b940017f0 con 0x7f0b980b4d60 2024-03-25T18:22:49.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0b9cde4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:49.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0b9cde4640 1 -- 172.21.15.22:0/2584603202 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0b94001c00 con 0x7f0b980b4d60 2024-03-25T18:22:49.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0b9cde4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0b9cde4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0b9cde4640 10 monclient: dump: 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0ba004a640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:49.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.267+0000 7f0ba004a640 4 set_mon_vals no callback set 2024-03-25T18:22:49.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.268+0000 7f0ba004a640 10 monclient: shutdown 2024-03-25T18:22:49.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.268+0000 7f0ba004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.268+0000 7f0ba004a640 1 -- 172.21.15.22:0/2584603202 >> v1:172.21.15.22:6789/0 conn(0x7f0b980b4d60 legacy=0x7f0b9813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.268+0000 7f0ba004a640 1 -- 172.21.15.22:0/2584603202 shutdown_connections 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.268+0000 7f0ba004a640 1 -- 172.21.15.22:0/2584603202 >> 172.21.15.22:0/2584603202 conn(0x7f0b980617a0 msgr2=0x7f0b98061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.268+0000 7f0ba004a640 1 -- 172.21.15.22:0/2584603202 shutdown_connections 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.268+0000 7f0ba004a640 1 -- 172.21.15.22:0/2584603202 wait complete. 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.269+0000 7f0ba004a640 10 monclient: build_initial_monmap 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.269+0000 7f0ba004a640 10 monclient: monmap: 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:49.270 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:49.271 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:49.271 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.270+0000 7f0ba004a640 1 librados: starting msgr at 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.270+0000 7f0ba004a640 1 librados: starting objecter 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.270+0000 7f0ba004a640 1 Processor -- start 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.270+0000 7f0ba004a640 1 -- start start 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.270+0000 7f0ba004a640 1 librados: setting wanted keys 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.270+0000 7f0ba004a640 1 librados: calling monclient init 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.270+0000 7f0ba004a640 10 monclient: init 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:49.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 10 monclient(hunting): picked mon.noname-a con 0x7f0b980b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 10 monclient(hunting): picked mon.noname-c con 0x7f0b980b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 10 monclient(hunting): picked mon.noname-b con 0x7f0b9814be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b98138210 con 0x7f0b980b1730 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b981375a0 con 0x7f0b9814be90 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b98138440 con 0x7f0b980b4d60 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:49.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.271+0000 7f0ba004a640 10 monclient(hunting): authenticate will time out at 7368.494025s 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.284+0000 7f0b9e5e7640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f0b9814be90 0x7f0b981e3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:38028/0 (socket says 172.21.15.22:38028) 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.284+0000 7f0b9e5e7640 1 -- 172.21.15.22:0/14637471 learned_addr learned my addr 172.21.15.22:0/14637471 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:49.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.287+0000 7f0b8e7fc640 10 client.?.objecter ms_handle_connect 0x7f0b9814be90 2024-03-25T18:22:49.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.287+0000 7f0b8e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.289+0000 7f0b8e7fc640 10 client.?.objecter ms_handle_connect 0x7f0b980b4d60 2024-03-25T18:22:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.289+0000 7f0b8e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.294+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134583617 0 0) 0x7f0b981375a0 con 0x7f0b9814be90 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.294+0000 7f0b8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.294+0000 7f0b8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.294+0000 7f0b8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.294+0000 7f0b8e7fc640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.295+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b7c004eb0 con 0x7f0b9814be90 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 10 client.?.objecter ms_handle_connect 0x7f0b980b1730 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849166001 0 0) 0x7f0b98138440 con 0x7f0b980b4d60 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:22:49.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.297+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b981375a0 con 0x7f0b980b4d60 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.313+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777264260 0 0) 0x7f0b98138210 con 0x7f0b980b1730 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.313+0000 7f0b8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.313+0000 7f0b8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.313+0000 7f0b8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.313+0000 7f0b8e7fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.313+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b98138440 con 0x7f0b980b1730 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.315+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 519392207 0 0) 0x7f0b7c004eb0 con 0x7f0b9814be90 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.315+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b98138210 con 0x7f0b9814be90 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.315+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0b90001c90 con 0x7f0b9814be90 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.315+0000 7f0b8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.315+0000 7f0b8e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.315+0000 7f0b8e7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:49.317 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:49.317 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:49.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.315+0000 7f0b8e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:49.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.317+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573567427 0 0) 0x7f0b981375a0 con 0x7f0b980b4d60 2024-03-25T18:22:49.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.317+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b7c004eb0 con 0x7f0b980b4d60 2024-03-25T18:22:49.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.318+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0b88001970 con 0x7f0b980b4d60 2024-03-25T18:22:49.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.318+0000 7f0b8e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.318+0000 7f0b8e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:49.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.318+0000 7f0b8e7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:49.318 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:49.319 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:49.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1238853956 0 0) 0x7f0b98138210 con 0x7f0b9814be90 2024-03-25T18:22:49.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 >> v1:172.21.15.22:6790/0 conn(0x7f0b980b4d60 legacy=0x7f0b9812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 >> v1:172.21.15.22:6789/0 conn(0x7f0b980b1730 legacy=0x7f0b98139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 1 monclient: found mon.b 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b981364b0 con 0x7f0b9814be90 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0b8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 10 monclient: _renew_subs 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b9813b470 con 0x7f0b9814be90 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 10 monclient: _renew_subs 2024-03-25T18:22:49.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b98136230 con 0x7f0b9814be90 2024-03-25T18:22:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.319+0000 7f0ba004a640 1 librados: init done 2024-03-25T18:22:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.320+0000 7f0ba004a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.320+0000 7f0ba004a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.320+0000 7f0ba004a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.320+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b980627b0 con 0x7f0b9814be90 2024-03-25T18:22:49.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b90003990 con 0x7f0b9814be90 2024-03-25T18:22:49.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:49.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f0b90003d60 con 0x7f0b9814be90 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8e7fc640 10 monclient: dump: 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:49.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8ffff640 4 set_mon_vals no callback set 2024-03-25T18:22:49.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.600+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f0b90003f80 con 0x7f0b9814be90 2024-03-25T18:22:49.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.601+0000 7f0b9dde6640 10 monclient: get_auth_request con 0x7f0b7c06def0 auth_method 0 2024-03-25T18:22:49.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.601+0000 7f0b8e7fc640 10 client.4384.objecter ms_handle_connect 0x7f0b7c06def0 2024-03-25T18:22:49.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.608+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f0b90054aa0 con 0x7f0b9814be90 2024-03-25T18:22:49.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.608+0000 7f0b8e7fc640 10 client.4384.objecter ms_dispatch 0x7f0b98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:49.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.608+0000 7f0b8e7fc640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:49.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.608+0000 7f0b8e7fc640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:49.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.608+0000 7f0b8e7fc640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T18:22:49.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.609+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b900505b0 con 0x7f0b9814be90 2024-03-25T18:22:49.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.609+0000 7f0b8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.609+0000 7f0b8e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:49.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.862+0000 7f0ba004a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:49.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.862+0000 7f0ba004a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:49.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.862+0000 7f0ba004a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:22:49.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.862+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b98136640 con 0x7f0b9814be90 2024-03-25T18:22:49.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.885+0000 7f0b8e7fc640 1 -- 172.21.15.22:0/14637471 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f0b900524e0 con 0x7f0b9814be90 2024-03-25T18:22:49.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.885+0000 7f0b8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:49.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.885+0000 7f0b8e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:49.886 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:49.886 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:49.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 10 librados: watch_flush enter 2024-03-25T18:22:49.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 10 librados: watch_flush exit 2024-03-25T18:22:49.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T18:22:49.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 10 client.4384.objecter successfully canceled tick 2024-03-25T18:22:49.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 >> v1:172.21.15.33:6800/215350958 conn(0x7f0b7c06def0 legacy=0x7f0b7c0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 10 monclient: shutdown 2024-03-25T18:22:49.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:49.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.887+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 >> v1:172.21.15.33:6789/0 conn(0x7f0b9814be90 legacy=0x7f0b981e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.888+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 shutdown_connections 2024-03-25T18:22:49.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.888+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 >> 172.21.15.22:0/14637471 conn(0x7f0b980617a0 msgr2=0x7f0b9813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:49.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.888+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 shutdown_connections 2024-03-25T18:22:49.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.888+0000 7f0ba004a640 1 -- 172.21.15.22:0/14637471 wait complete. 2024-03-25T18:22:49.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:49.888+0000 7f0ba004a640 1 librados: shutdown 2024-03-25T18:22:49.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:52.547 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:55.209 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.562 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:55.224 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.608 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:55.270 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.609 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:03:55.271 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.619 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:55.281 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.631 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:55.249 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.634 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:55.252 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.646 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:03:55.308 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:52.663 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:03:55.281 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:56.708 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.829+0000 7f54563cf640 10 monclient: get_monmap_and_config 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.829+0000 7f54563cf640 10 monclient: build_initial_monmap 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.829+0000 7f54563cf640 10 monclient: monmap: 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:22:56.830560+0000 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:22:56.830560+0000 2024-03-25T18:22:56.830 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 1 Processor -- start 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 1 -- start start 2024-03-25T18:22:56.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 10 monclient: init 2024-03-25T18:22:56.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:56.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:56.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 10 monclient(hunting): picked mon.noname-a con 0x7f54500a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:56.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 10 monclient(hunting): picked mon.noname-b con 0x7f545013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:56.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 10 monclient(hunting): picked mon.noname-c con 0x7f54501430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:56.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.830+0000 7f54563cf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450142520 con 0x7f54500a98f0 2024-03-25T18:22:56.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.831+0000 7f54563cf640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b28a0 con 0x7f54501430c0 2024-03-25T18:22:56.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.831+0000 7f54563cf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54500b2a30 con 0x7f545013f710 2024-03-25T18:22:56.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.831+0000 7f54563cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:56.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.831+0000 7f54563cf640 10 monclient(hunting): authenticate will time out at 7376.054671s 2024-03-25T18:22:56.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.834+0000 7f545496c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f54501430c0 0x7f545014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:40078/0 (socket says 172.21.15.22:40078) 2024-03-25T18:22:56.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.834+0000 7f545496c640 1 -- 172.21.15.22:0/1415966396 learned_addr learned my addr 172.21.15.22:0/1415966396 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.928+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247500642 0 0) 0x7f5450142520 con 0x7f54500a98f0 2024-03-25T18:22:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.928+0000 7f544effd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.928+0000 7f544effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.928+0000 7f544effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.928+0000 7f544effd640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:22:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.928+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5438003520 con 0x7f54500a98f0 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2728904899 0 0) 0x7f5438003520 con 0x7f54500a98f0 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 >> v1:172.21.15.22:6790/0 conn(0x7f545013f710 legacy=0x7f54501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 >> v1:172.21.15.33:6789/0 conn(0x7f54501430c0 legacy=0x7f545014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 1 monclient: found mon.noname-a 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54500b44c0 con 0x7f54500a98f0 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 10 monclient: _finish_auth 0 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5440001690 con 0x7f54500a98f0 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f544effd640 10 monclient: dump: 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f54563cf640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:22:56.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.944+0000 7f54563cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:56.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54400017f0 con 0x7f54500a98f0 2024-03-25T18:22:56.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f544effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:56.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f544effd640 1 -- 172.21.15.22:0/1415966396 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5440001c00 con 0x7f54500a98f0 2024-03-25T18:22:56.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f54563cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:56.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f54563cf640 4 set_mon_vals no callback set 2024-03-25T18:22:56.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f544effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f544effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f544effd640 10 monclient: dump: 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:56.958 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f54563cf640 10 monclient: shutdown 2024-03-25T18:22:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f54563cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f54563cf640 1 -- 172.21.15.22:0/1415966396 >> v1:172.21.15.22:6789/0 conn(0x7f54500a98f0 legacy=0x7f54500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.956+0000 7f54563cf640 1 -- 172.21.15.22:0/1415966396 shutdown_connections 2024-03-25T18:22:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.957+0000 7f54563cf640 1 -- 172.21.15.22:0/1415966396 >> 172.21.15.22:0/1415966396 conn(0x7f54500b7ea0 msgr2=0x7f54500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.957+0000 7f54563cf640 1 -- 172.21.15.22:0/1415966396 shutdown_connections 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.957+0000 7f54563cf640 1 -- 172.21.15.22:0/1415966396 wait complete. 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 10 monclient: build_initial_monmap 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 10 monclient: monmap: 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 1 librados: starting msgr at 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 1 librados: starting objecter 2024-03-25T18:22:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 1 Processor -- start 2024-03-25T18:22:56.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 1 -- start start 2024-03-25T18:22:56.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 1 librados: setting wanted keys 2024-03-25T18:22:56.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 1 librados: calling monclient init 2024-03-25T18:22:56.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.958+0000 7f54563cf640 10 monclient: init 2024-03-25T18:22:56.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 10 monclient(hunting): picked mon.noname-c con 0x7f54500a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 10 monclient(hunting): picked mon.noname-b con 0x7f545013f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 10 monclient(hunting): picked mon.noname-a con 0x7f54501430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5450139420 con 0x7f54501430c0 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545006ee70 con 0x7f545013f710 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f545006f150 con 0x7f54500a98f0 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:56.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.959+0000 7f54563cf640 10 monclient(hunting): authenticate will time out at 7376.183045s 2024-03-25T18:22:56.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.965+0000 7f545496c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f54501430c0 0x7f54501e6130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:54454/0 (socket says 172.21.15.22:54454) 2024-03-25T18:22:56.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.965+0000 7f545496c640 1 -- 172.21.15.22:0/2386530128 learned_addr learned my addr 172.21.15.22:0/2386530128 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:22:56.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.967+0000 7f544cff9640 10 client.?.objecter ms_handle_connect 0x7f54501430c0 2024-03-25T18:22:56.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.967+0000 7f544cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:56.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.984+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528966841 0 0) 0x7f5450139420 con 0x7f54501430c0 2024-03-25T18:22:56.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.984+0000 7f544cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.984+0000 7f544cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.984+0000 7f544cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.984+0000 7f544cff9640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:22:56.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.984+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f543c004eb0 con 0x7f54501430c0 2024-03-25T18:22:56.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.998+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3183444124 0 0) 0x7f543c004eb0 con 0x7f54501430c0 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.998+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5450139420 con 0x7f54501430c0 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.998+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5448001b10 con 0x7f54501430c0 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.998+0000 7f544cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.998+0000 7f544cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.998+0000 7f544cff9640 10 monclient(hunting): dump: 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:57.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:56.998+0000 7f544cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:57.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.000+0000 7f544cff9640 10 client.?.objecter ms_handle_connect 0x7f545013f710 2024-03-25T18:22:57.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.000+0000 7f544cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:57.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.003+0000 7f544cff9640 10 client.?.objecter ms_handle_connect 0x7f54500a98f0 2024-03-25T18:22:57.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.003+0000 7f544cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:57.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2438114882 0 0) 0x7f5450139420 con 0x7f54501430c0 2024-03-25T18:22:57.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 >> v1:172.21.15.22:6790/0 conn(0x7f54500a98f0 legacy=0x7f545014d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 >> v1:172.21.15.33:6789/0 conn(0x7f545013f710 legacy=0x7f545014df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 1 monclient: found mon.a 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450139140 con 0x7f54501430c0 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f544cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 10 monclient: _renew_subs 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545006dea0 con 0x7f54501430c0 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 10 monclient: _renew_subs 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:57.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54500b7180 con 0x7f54501430c0 2024-03-25T18:22:57.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.018+0000 7f54563cf640 1 librados: init done 2024-03-25T18:22:57.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.019+0000 7f54563cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:57.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.019+0000 7f54563cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:57.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.019+0000 7f54563cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:57.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.019+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f545006e090 con 0x7f54501430c0 2024-03-25T18:22:57.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.332+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54480020c0 con 0x7f54501430c0 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.332+0000 7f544cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.332+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5448004d40 con 0x7f54501430c0 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.332+0000 7f544cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.332+0000 7f544cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.332+0000 7f544cff9640 10 monclient: dump: 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.332+0000 7f544e7fc640 4 set_mon_vals no callback set 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.333+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f5448004f60 con 0x7f54501430c0 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.334+0000 7f544ffff640 10 monclient: get_auth_request con 0x7f543c06dc20 auth_method 0 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.334+0000 7f544cff9640 10 client.4389.objecter ms_handle_connect 0x7f543c06dc20 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.338+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f5448054b80 con 0x7f54501430c0 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.338+0000 7f544cff9640 10 client.4389.objecter ms_dispatch 0x7f5450000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.338+0000 7f544cff9640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.338+0000 7f544cff9640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.339+0000 7f544cff9640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.339+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5448050690 con 0x7f54501430c0 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.339+0000 7f544cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:57.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.339+0000 7f544cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:57.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.595+0000 7f54563cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.595+0000 7f54563cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.595+0000 7f54563cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:22:57.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.595+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f545014d510 con 0x7f54501430c0 2024-03-25T18:22:57.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.652+0000 7f544cff9640 1 -- 172.21.15.22:0/2386530128 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f545014d510 con 0x7f54501430c0 2024-03-25T18:22:57.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.652+0000 7f544cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.652+0000 7f544cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:57.654 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:22:57.654 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 10 librados: watch_flush enter 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 10 librados: watch_flush exit 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 10 client.4389.objecter successfully canceled tick 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 >> v1:172.21.15.33:6800/215350958 conn(0x7f543c06dc20 legacy=0x7f543c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 10 monclient: shutdown 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:57.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 >> v1:172.21.15.22:6789/0 conn(0x7f54501430c0 legacy=0x7f54501e6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 shutdown_connections 2024-03-25T18:22:57.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 >> 172.21.15.22:0/2386530128 conn(0x7f54500b7ea0 msgr2=0x7f5450074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:57.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 shutdown_connections 2024-03-25T18:22:57.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 1 -- 172.21.15.22:0/2386530128 wait complete. 2024-03-25T18:22:57.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:22:57.655+0000 7f54563cf640 1 librados: shutdown 2024-03-25T18:22:57.669 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:04.475 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.590+0000 7fbfa233b640 10 monclient: get_monmap_and_config 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.590+0000 7fbfa233b640 10 monclient: build_initial_monmap 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.590+0000 7fbfa233b640 10 monclient: monmap: 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:04.592056+0000 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:04.592056+0000 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:04.592 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:04.593 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:04.593 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:04.593 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:04.593 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.591+0000 7fbfa233b640 1 Processor -- start 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.591+0000 7fbfa233b640 1 -- start start 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.591+0000 7fbfa233b640 10 monclient: init 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 10 monclient(hunting): picked mon.noname-c con 0x7fbf9c0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 10 monclient(hunting): picked mon.noname-b con 0x7fbf9c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c142520 con 0x7fbf9c1430c0 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0b28a0 con 0x7fbf9c0a98f0 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0b2a30 con 0x7fbf9c13f710 2024-03-25T18:23:04.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:04.595 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.592+0000 7fbfa233b640 10 monclient(hunting): authenticate will time out at 7383.816087s 2024-03-25T18:23:04.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.604+0000 7fbf9bfff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fbf9c0a98f0 0x7fbf9c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:37712/0 (socket says 172.21.15.22:37712) 2024-03-25T18:23:04.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.604+0000 7fbf9bfff640 1 -- 172.21.15.22:0/3802239381 learned_addr learned my addr 172.21.15.22:0/3802239381 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:04.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.617+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610354292 0 0) 0x7fbf9c0b28a0 con 0x7fbf9c0a98f0 2024-03-25T18:23:04.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.617+0000 7fbf9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.617+0000 7fbf9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.617+0000 7fbf9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.617+0000 7fbf9affd640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:23:04.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.617+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf84003520 con 0x7fbf9c0a98f0 2024-03-25T18:23:04.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.627+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919242245 0 0) 0x7fbf9c0b2a30 con 0x7fbf9c13f710 2024-03-25T18:23:04.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.627+0000 7fbf9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.627+0000 7fbf9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.627+0000 7fbf9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.627+0000 7fbf9affd640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:23:04.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.627+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c0b28a0 con 0x7fbf9c13f710 2024-03-25T18:23:04.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.631+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221713426 0 0) 0x7fbf9c142520 con 0x7fbf9c1430c0 2024-03-25T18:23:04.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.631+0000 7fbf9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.631+0000 7fbf9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.631+0000 7fbf9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.631+0000 7fbf9affd640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:23:04.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.631+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c0b2a30 con 0x7fbf9c1430c0 2024-03-25T18:23:04.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 143709572 0 0) 0x7fbf9c0b28a0 con 0x7fbf9c13f710 2024-03-25T18:23:04.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 >> v1:172.21.15.33:6789/0 conn(0x7fbf9c0a98f0 legacy=0x7fbf9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 >> v1:172.21.15.22:6789/0 conn(0x7fbf9c1430c0 legacy=0x7fbf9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 1 monclient: found mon.noname-b 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0b44c0 con 0x7fbf9c13f710 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 10 monclient: _finish_auth 0 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf90001670 con 0x7fbf9c13f710 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbfa233b640 5 monclient: authenticate success, global_id 4394 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbfa233b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbf9affd640 10 monclient: dump: 2024-03-25T18:23:04.636 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:04.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.634+0000 7fbfa233b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf900017d0 con 0x7fbf9c13f710 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbf9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbf9affd640 1 -- 172.21.15.22:0/3802239381 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf90001be0 con 0x7fbf9c13f710 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbfa233b640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbfa233b640 4 set_mon_vals no callback set 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbf9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbf9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.658+0000 7fbf9affd640 10 monclient: dump: 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:04.660 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.661 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:04.661 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:04.661 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:04.661 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:04.661 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.659+0000 7fbfa233b640 10 monclient: shutdown 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.659+0000 7fbfa233b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.659+0000 7fbfa233b640 1 -- 172.21.15.22:0/3802239381 >> v1:172.21.15.22:6790/0 conn(0x7fbf9c13f710 legacy=0x7fbf9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.660+0000 7fbfa233b640 1 -- 172.21.15.22:0/3802239381 shutdown_connections 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.660+0000 7fbfa233b640 1 -- 172.21.15.22:0/3802239381 >> 172.21.15.22:0/3802239381 conn(0x7fbf9c0b7ea0 msgr2=0x7fbf9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.660+0000 7fbfa233b640 1 -- 172.21.15.22:0/3802239381 shutdown_connections 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.660+0000 7fbfa233b640 1 -- 172.21.15.22:0/3802239381 wait complete. 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.661+0000 7fbfa233b640 10 monclient: build_initial_monmap 2024-03-25T18:23:04.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.661+0000 7fbfa233b640 10 monclient: monmap: 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:04.663 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.661+0000 7fbfa233b640 1 librados: starting msgr at 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.661+0000 7fbfa233b640 1 librados: starting objecter 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.661+0000 7fbfa233b640 1 Processor -- start 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 1 -- start start 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 1 librados: setting wanted keys 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 1 librados: calling monclient init 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient: init 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient(hunting): picked mon.noname-c con 0x7fbf9c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient(hunting): picked mon.noname-b con 0x7fbf9c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c138700 con 0x7fbf9c0a98f0 2024-03-25T18:23:04.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c138b90 con 0x7fbf9c1430c0 2024-03-25T18:23:04.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0b16d0 con 0x7fbf9c13f710 2024-03-25T18:23:04.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:04.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.662+0000 7fbfa233b640 10 monclient(hunting): authenticate will time out at 7383.886534s 2024-03-25T18:23:04.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.675+0000 7fbfa08d8640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fbf9c1430c0 0x7fbf9c1e1dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:37746/0 (socket says 172.21.15.22:37746) 2024-03-25T18:23:04.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.675+0000 7fbfa08d8640 1 -- 172.21.15.22:0/973805559 learned_addr learned my addr 172.21.15.22:0/973805559 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:04.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.676+0000 7fbf98ff9640 10 client.?.objecter ms_handle_connect 0x7fbf9c1430c0 2024-03-25T18:23:04.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.677+0000 7fbf98ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:04.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.681+0000 7fbf98ff9640 10 client.?.objecter ms_handle_connect 0x7fbf9c0a98f0 2024-03-25T18:23:04.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.681+0000 7fbf98ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:04.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.684+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627247521 0 0) 0x7fbf9c138b90 con 0x7fbf9c1430c0 2024-03-25T18:23:04.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.684+0000 7fbf98ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.684+0000 7fbf98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.684+0000 7fbf98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.684+0000 7fbf98ff9640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:23:04.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.684+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf88004eb0 con 0x7fbf9c1430c0 2024-03-25T18:23:04.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.686+0000 7fbf98ff9640 10 client.?.objecter ms_handle_connect 0x7fbf9c13f710 2024-03-25T18:23:04.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.686+0000 7fbf98ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:04.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.705+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1307686553 0 0) 0x7fbf9c138700 con 0x7fbf9c0a98f0 2024-03-25T18:23:04.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.705+0000 7fbf98ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.705+0000 7fbf98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.705+0000 7fbf98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.705+0000 7fbf98ff9640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:23:04.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.705+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c138b90 con 0x7fbf9c0a98f0 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.710+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1314991700 0 0) 0x7fbf88004eb0 con 0x7fbf9c1430c0 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.710+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf9c138700 con 0x7fbf9c1430c0 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.710+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf94001c70 con 0x7fbf9c1430c0 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.710+0000 7fbf98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.710+0000 7fbf98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.710+0000 7fbf98ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:04.712 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:04.713 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.713 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:04.713 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:04.713 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:04.713 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:04.713 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:04.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.710+0000 7fbf98ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:04.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.721+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3560605559 0 0) 0x7fbf9c138700 con 0x7fbf9c1430c0 2024-03-25T18:23:04.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.721+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 >> v1:172.21.15.22:6790/0 conn(0x7fbf9c13f710 legacy=0x7fbf9c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 >> v1:172.21.15.22:6789/0 conn(0x7fbf9c0a98f0 legacy=0x7fbf9c137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 1 monclient: found mon.b 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c138050 con 0x7fbf9c1430c0 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbf98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 10 monclient: _renew_subs 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c1382a0 con 0x7fbf9c1430c0 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 10 monclient: _renew_subs 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:04.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c1db170 con 0x7fbf9c1430c0 2024-03-25T18:23:04.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.722+0000 7fbfa233b640 1 librados: init done 2024-03-25T18:23:04.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.723+0000 7fbfa233b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:04.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.723+0000 7fbfa233b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:04.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.723+0000 7fbfa233b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:04.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:04.723+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c1384e0 con 0x7fbf9c1430c0 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.023+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf94003970 con 0x7fbf9c1430c0 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.023+0000 7fbf98ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.023+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fbf94003d80 con 0x7fbf9c1430c0 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.023+0000 7fbf98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.023+0000 7fbf98ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.023+0000 7fbf98ff9640 10 monclient: dump: 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:05.025 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:05.026 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:05.026 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:05.026 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:05.026 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:05.026 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:05.026 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:05.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.023+0000 7fbf9a7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:05.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.024+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fbf94003fa0 con 0x7fbf9c1430c0 2024-03-25T18:23:05.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.024+0000 7fbf9bfff640 10 monclient: get_auth_request con 0x7fbf8806dd10 auth_method 0 2024-03-25T18:23:05.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.025+0000 7fbf98ff9640 10 client.4396.objecter ms_handle_connect 0x7fbf8806dd10 2024-03-25T18:23:05.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.030+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fbf94054ac0 con 0x7fbf9c1430c0 2024-03-25T18:23:05.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.030+0000 7fbf98ff9640 10 client.4396.objecter ms_dispatch 0x7fbf9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:05.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.030+0000 7fbf98ff9640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:05.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.030+0000 7fbf98ff9640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:05.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.030+0000 7fbf98ff9640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T18:23:05.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.031+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf940505d0 con 0x7fbf9c1430c0 2024-03-25T18:23:05.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.031+0000 7fbf98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:05.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.031+0000 7fbf98ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:05.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.288+0000 7fbfa233b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.288+0000 7fbfa233b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.288+0000 7fbfa233b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:05.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.288+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf9c0b3ef0 con 0x7fbf9c1430c0 2024-03-25T18:23:05.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.426+0000 7fbf98ff9640 1 -- 172.21.15.22:0/973805559 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fbf94052500 con 0x7fbf9c1430c0 2024-03-25T18:23:05.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.426+0000 7fbf98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.426+0000 7fbf98ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:05.428 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:05.428 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:05.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 10 librados: watch_flush enter 2024-03-25T18:23:05.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 10 librados: watch_flush exit 2024-03-25T18:23:05.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T18:23:05.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 10 client.4396.objecter successfully canceled tick 2024-03-25T18:23:05.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 >> v1:172.21.15.33:6800/215350958 conn(0x7fbf8806dd10 legacy=0x7fbf880701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 10 monclient: shutdown 2024-03-25T18:23:05.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:05.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 >> v1:172.21.15.33:6789/0 conn(0x7fbf9c1430c0 legacy=0x7fbf9c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 shutdown_connections 2024-03-25T18:23:05.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 >> 172.21.15.22:0/973805559 conn(0x7fbf9c0b7ea0 msgr2=0x7fbf9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:05.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 shutdown_connections 2024-03-25T18:23:05.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 1 -- 172.21.15.22:0/973805559 wait complete. 2024-03-25T18:23:05.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:05.428+0000 7fbfa233b640 1 librados: shutdown 2024-03-25T18:23:05.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:12.248 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:12.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.360+0000 7f15be754640 10 monclient: get_monmap_and_config 2024-03-25T18:23:12.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.360+0000 7f15be754640 10 monclient: build_initial_monmap 2024-03-25T18:23:12.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.360+0000 7f15be754640 10 monclient: monmap: 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:12.361817+0000 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:12.361817+0000 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:12.362 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.361+0000 7f15be754640 1 Processor -- start 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.361+0000 7f15be754640 1 -- start start 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.361+0000 7f15be754640 10 monclient: init 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 10 monclient(hunting): picked mon.noname-a con 0x7f15b80b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 10 monclient(hunting): picked mon.noname-b con 0x7f15b80b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 10 monclient(hunting): picked mon.noname-c con 0x7f15b814be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80af940 con 0x7f15b80b4d60 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80b0a70 con 0x7f15b814be90 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b80b0d50 con 0x7f15b80b1730 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:12.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.362+0000 7f15be754640 10 monclient(hunting): authenticate will time out at 7391.585999s 2024-03-25T18:23:12.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.373+0000 7f15bccf1640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f15b814be90 0x7f15b814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:42228/0 (socket says 172.21.15.22:42228) 2024-03-25T18:23:12.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.373+0000 7f15bccf1640 1 -- 172.21.15.22:0/3054888911 learned_addr learned my addr 172.21.15.22:0/3054888911 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:12.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.386+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865556188 0 0) 0x7f15b80b0a70 con 0x7f15b814be90 2024-03-25T18:23:12.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.386+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.386+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.386+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.386+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:23:12.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.386+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a0003520 con 0x7f15b814be90 2024-03-25T18:23:12.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.395+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414302753 0 0) 0x7f15b80b0d50 con 0x7f15b80b1730 2024-03-25T18:23:12.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.395+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.395+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.395+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.395+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:23:12.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.395+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b80b0a70 con 0x7f15b80b1730 2024-03-25T18:23:12.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.406+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498376771 0 0) 0x7f15b80af940 con 0x7f15b80b4d60 2024-03-25T18:23:12.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.406+0000 7f15b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.406+0000 7f15b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.406+0000 7f15b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.406+0000 7f15b6ffd640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:23:12.407 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.406+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b80b0d50 con 0x7f15b80b4d60 2024-03-25T18:23:12.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 11709536 0 0) 0x7f15a0003520 con 0x7f15b814be90 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 >> v1:172.21.15.22:6790/0 conn(0x7f15b80b1730 legacy=0x7f15b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 >> v1:172.21.15.22:6789/0 conn(0x7f15b80b4d60 legacy=0x7f15b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b80b1030 con 0x7f15b814be90 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f15b0001a10 con 0x7f15b814be90 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:12.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15b6ffd640 10 monclient: dump: 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:12.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15be754640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:23:12.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.411+0000 7f15be754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15b0001b70 con 0x7f15b814be90 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15b6ffd640 1 -- 172.21.15.22:0/3054888911 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f15b0001f80 con 0x7f15b814be90 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 4 set_mon_vals no callback set 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:12.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15b6ffd640 10 monclient: dump: 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:12.426 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:12.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 10 monclient: shutdown 2024-03-25T18:23:12.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 1 -- 172.21.15.22:0/3054888911 >> v1:172.21.15.33:6789/0 conn(0x7f15b814be90 legacy=0x7f15b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 1 -- 172.21.15.22:0/3054888911 shutdown_connections 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 1 -- 172.21.15.22:0/3054888911 >> 172.21.15.22:0/3054888911 conn(0x7f15b80617a0 msgr2=0x7f15b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.424+0000 7f15be754640 1 -- 172.21.15.22:0/3054888911 shutdown_connections 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.425+0000 7f15be754640 1 -- 172.21.15.22:0/3054888911 wait complete. 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.425+0000 7f15be754640 10 monclient: build_initial_monmap 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.425+0000 7f15be754640 10 monclient: monmap: 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:12.428 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.426+0000 7f15be754640 1 librados: starting msgr at 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.426+0000 7f15be754640 1 librados: starting objecter 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.426+0000 7f15be754640 1 Processor -- start 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.426+0000 7f15be754640 1 -- start start 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.426+0000 7f15be754640 1 librados: setting wanted keys 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.426+0000 7f15be754640 1 librados: calling monclient init 2024-03-25T18:23:12.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.426+0000 7f15be754640 10 monclient: init 2024-03-25T18:23:12.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:12.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:12.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 10 monclient(hunting): picked mon.noname-b con 0x7f15b80b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:12.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 10 monclient(hunting): picked mon.noname-c con 0x7f15b80b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:12.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 10 monclient(hunting): picked mon.noname-a con 0x7f15b814be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:12.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b8135b00 con 0x7f15b814be90 2024-03-25T18:23:12.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b8136270 con 0x7f15b80b1730 2024-03-25T18:23:12.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15b8137c80 con 0x7f15b80b4d60 2024-03-25T18:23:12.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:12.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.427+0000 7f15be754640 10 monclient(hunting): authenticate will time out at 7391.651222s 2024-03-25T18:23:12.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.438+0000 7f15b7fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f15b80b1730 0x7f15b812fe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:42274/0 (socket says 172.21.15.22:42274) 2024-03-25T18:23:12.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.438+0000 7f15b7fff640 1 -- 172.21.15.22:0/3259744179 learned_addr learned my addr 172.21.15.22:0/3259744179 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:12.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.438+0000 7f15b4ff9640 10 client.?.objecter ms_handle_connect 0x7f15b80b1730 2024-03-25T18:23:12.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.439+0000 7f15b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.440+0000 7f15b4ff9640 10 client.?.objecter ms_handle_connect 0x7f15b80b4d60 2024-03-25T18:23:12.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.440+0000 7f15b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.444+0000 7f15b4ff9640 10 client.?.objecter ms_handle_connect 0x7f15b814be90 2024-03-25T18:23:12.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.444+0000 7f15b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.452+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296322357 0 0) 0x7f15b8136270 con 0x7f15b80b1730 2024-03-25T18:23:12.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.452+0000 7f15b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.452+0000 7f15b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.452+0000 7f15b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.452+0000 7f15b4ff9640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:23:12.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.452+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a4004eb0 con 0x7f15b80b1730 2024-03-25T18:23:12.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.453+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594607949 0 0) 0x7f15b8135b00 con 0x7f15b814be90 2024-03-25T18:23:12.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.453+0000 7f15b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.453+0000 7f15b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.453+0000 7f15b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.453+0000 7f15b4ff9640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:23:12.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.453+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b8136270 con 0x7f15b814be90 2024-03-25T18:23:12.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.458+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251853759 0 0) 0x7f15b8137c80 con 0x7f15b80b4d60 2024-03-25T18:23:12.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.458+0000 7f15b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.458+0000 7f15b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.458+0000 7f15b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.458+0000 7f15b4ff9640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:23:12.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.458+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15b8135b00 con 0x7f15b80b4d60 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.461+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1343755952 0 0) 0x7f15a4004eb0 con 0x7f15b80b1730 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.461+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15b8137c80 con 0x7f15b80b1730 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.461+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f15a8001a10 con 0x7f15b80b1730 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.461+0000 7f15b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.461+0000 7f15b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.461+0000 7f15b4ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.463 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.464 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:12.464 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:12.464 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:12.464 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:12.464 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:12.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.461+0000 7f15b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.464+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970968684 0 0) 0x7f15b8136270 con 0x7f15b814be90 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.464+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a4004eb0 con 0x7f15b814be90 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.464+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f15b0001e80 con 0x7f15b814be90 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.464+0000 7f15b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.464+0000 7f15b4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.464+0000 7f15b4ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:12.465 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:12.466 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1917558093 0 0) 0x7f15a4004eb0 con 0x7f15b814be90 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 >> v1:172.21.15.22:6790/0 conn(0x7f15b80b4d60 legacy=0x7f15b81dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 >> v1:172.21.15.33:6789/0 conn(0x7f15b80b1730 legacy=0x7f15b812fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 1 monclient: found mon.a 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b8135d50 con 0x7f15b814be90 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:12.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 10 monclient: _renew_subs 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b81369a0 con 0x7f15b814be90 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 10 monclient: _renew_subs 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b8130750 con 0x7f15b814be90 2024-03-25T18:23:12.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.478+0000 7f15be754640 1 librados: init done 2024-03-25T18:23:12.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.479+0000 7f15be754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.479+0000 7f15be754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.479+0000 7f15be754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:23:12.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.479+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b81401f0 con 0x7f15b814be90 2024-03-25T18:23:12.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.805+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15b0001fe0 con 0x7f15b814be90 2024-03-25T18:23:12.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.805+0000 7f15b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:12.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.805+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f15b0004fb0 con 0x7f15b814be90 2024-03-25T18:23:12.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.805+0000 7f15b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.805+0000 7f15b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:12.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.805+0000 7f15b4ff9640 10 monclient: dump: 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:12.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.805+0000 7f15b67fc640 4 set_mon_vals no callback set 2024-03-25T18:23:12.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.806+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f15b001af80 con 0x7f15b814be90 2024-03-25T18:23:12.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.806+0000 7f15b7fff640 10 monclient: get_auth_request con 0x7f15a406df70 auth_method 0 2024-03-25T18:23:12.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.807+0000 7f15b4ff9640 10 client.4401.objecter ms_handle_connect 0x7f15a406df70 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f15b0054a20 con 0x7f15b814be90 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 10 client.4401.objecter ms_dispatch 0x7f15b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15b0050530 con 0x7f15b814be90 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:12.815+0000 7f15b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:13.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.032+0000 7f15be754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:13.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.032+0000 7f15be754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:13.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.032+0000 7f15be754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:23:13.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.032+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15b80627b0 con 0x7f15b814be90 2024-03-25T18:23:13.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.111+0000 7f15b4ff9640 1 -- 172.21.15.22:0/3259744179 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f15b0052460 con 0x7f15b814be90 2024-03-25T18:23:13.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.111+0000 7f15b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:13.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.111+0000 7f15b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:13.112 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:13.112 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 10 librados: watch_flush enter 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 10 librados: watch_flush exit 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 10 client.4401.objecter successfully canceled tick 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 >> v1:172.21.15.33:6800/215350958 conn(0x7f15a406df70 legacy=0x7f15a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 10 monclient: shutdown 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 >> v1:172.21.15.22:6789/0 conn(0x7f15b814be90 legacy=0x7f15b81e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 shutdown_connections 2024-03-25T18:23:13.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 >> 172.21.15.22:0/3259744179 conn(0x7f15b80617a0 msgr2=0x7f15b813c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:13.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 shutdown_connections 2024-03-25T18:23:13.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 1 -- 172.21.15.22:0/3259744179 wait complete. 2024-03-25T18:23:13.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:13.113+0000 7f15be754640 1 librados: shutdown 2024-03-25T18:23:13.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:19.932 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:20.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.046+0000 7fb914084640 10 monclient: get_monmap_and_config 2024-03-25T18:23:20.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.046+0000 7fb914084640 10 monclient: build_initial_monmap 2024-03-25T18:23:20.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.046+0000 7fb914084640 10 monclient: monmap: 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:20.047560+0000 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:20.047560+0000 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:20.048 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.046+0000 7fb914084640 1 Processor -- start 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 1 -- start start 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient: init 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient(hunting): picked mon.noname-a con 0x7fb90c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient(hunting): picked mon.noname-b con 0x7fb90c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:20.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient(hunting): picked mon.noname-c con 0x7fb90c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:20.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c142520 con 0x7fb90c0a98f0 2024-03-25T18:23:20.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c0b28a0 con 0x7fb90c1430c0 2024-03-25T18:23:20.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c0b2a30 con 0x7fb90c13f710 2024-03-25T18:23:20.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:20.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.047+0000 7fb914084640 10 monclient(hunting): authenticate will time out at 7399.271491s 2024-03-25T18:23:20.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.056+0000 7fb911e20640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb90c0a98f0 0x7fb90c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58946/0 (socket says 172.21.15.22:58946) 2024-03-25T18:23:20.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.056+0000 7fb911e20640 1 -- 172.21.15.22:0/2765257431 learned_addr learned my addr 172.21.15.22:0/2765257431 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:20.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.075+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435358909 0 0) 0x7fb90c0b28a0 con 0x7fb90c1430c0 2024-03-25T18:23:20.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.075+0000 7fb910e1e640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.075+0000 7fb910e1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.075+0000 7fb910e1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.075+0000 7fb910e1e640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:23:20.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.075+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f8003520 con 0x7fb90c1430c0 2024-03-25T18:23:20.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.081+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904881896 0 0) 0x7fb90c142520 con 0x7fb90c0a98f0 2024-03-25T18:23:20.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.081+0000 7fb910e1e640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.081+0000 7fb910e1e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.081+0000 7fb910e1e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.081+0000 7fb910e1e640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:23:20.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.081+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb90c0b28a0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 897870712 0 0) 0x7fb8f8003520 con 0x7fb90c1430c0 2024-03-25T18:23:20.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 >> v1:172.21.15.22:6790/0 conn(0x7fb90c13f710 legacy=0x7fb90c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 >> v1:172.21.15.22:6789/0 conn(0x7fb90c0a98f0 legacy=0x7fb90c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 10 monclient: _finish_hunting 0 2024-03-25T18:23:20.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 1 monclient: found mon.noname-c 2024-03-25T18:23:20.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:20.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c0b44c0 con 0x7fb90c1430c0 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 10 monclient: _finish_auth 0 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 10 monclient: _check_auth_tickets 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb904001a10 con 0x7fb90c1430c0 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb910e1e640 10 monclient: dump: 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:20.089 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:20.090 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb914084640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:23:20.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.086+0000 7fb914084640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb904001b70 con 0x7fb90c1430c0 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb910e1e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb910e1e640 1 -- 172.21.15.22:0/2765257431 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb904001f80 con 0x7fb90c1430c0 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb910e1e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb910e1e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb910e1e640 10 monclient: dump: 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.108 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:20.109 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:20.109 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:20.109 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:20.109 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb914084640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:20.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.106+0000 7fb914084640 4 set_mon_vals no callback set 2024-03-25T18:23:20.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.107+0000 7fb914084640 10 monclient: shutdown 2024-03-25T18:23:20.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.107+0000 7fb914084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:20.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.107+0000 7fb914084640 1 -- 172.21.15.22:0/2765257431 >> v1:172.21.15.33:6789/0 conn(0x7fb90c1430c0 legacy=0x7fb90c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.107+0000 7fb914084640 1 -- 172.21.15.22:0/2765257431 shutdown_connections 2024-03-25T18:23:20.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.107+0000 7fb914084640 1 -- 172.21.15.22:0/2765257431 >> 172.21.15.22:0/2765257431 conn(0x7fb90c0b7ea0 msgr2=0x7fb90c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:20.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.108+0000 7fb914084640 1 -- 172.21.15.22:0/2765257431 shutdown_connections 2024-03-25T18:23:20.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.108+0000 7fb914084640 1 -- 172.21.15.22:0/2765257431 wait complete. 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.108+0000 7fb914084640 10 monclient: build_initial_monmap 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.108+0000 7fb914084640 10 monclient: monmap: 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.109+0000 7fb914084640 1 librados: starting msgr at 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.109+0000 7fb914084640 1 librados: starting objecter 2024-03-25T18:23:20.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.109+0000 7fb914084640 1 Processor -- start 2024-03-25T18:23:20.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.109+0000 7fb914084640 1 -- start start 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.109+0000 7fb914084640 1 librados: setting wanted keys 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.109+0000 7fb914084640 1 librados: calling monclient init 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.109+0000 7fb914084640 10 monclient: init 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 10 monclient(hunting): picked mon.noname-c con 0x7fb90c0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 10 monclient(hunting): picked mon.noname-b con 0x7fb90c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 10 monclient(hunting): picked mon.noname-a con 0x7fb90c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c1daca0 con 0x7fb90c1430c0 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c138ab0 con 0x7fb90c13f710 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c0aba90 con 0x7fb90c0a98f0 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:20.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.110+0000 7fb914084640 10 monclient(hunting): authenticate will time out at 7399.334208s 2024-03-25T18:23:20.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.114+0000 7fb912621640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb90c1430c0 0x7fb90c1d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58972/0 (socket says 172.21.15.22:58972) 2024-03-25T18:23:20.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.114+0000 7fb912621640 1 -- 172.21.15.22:0/1651168043 learned_addr learned my addr 172.21.15.22:0/1651168043 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:20.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.114+0000 7fb9027fc640 10 client.?.objecter ms_handle_connect 0x7fb90c1430c0 2024-03-25T18:23:20.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.114+0000 7fb9027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:20.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.122+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745820804 0 0) 0x7fb90c1daca0 con 0x7fb90c1430c0 2024-03-25T18:23:20.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.122+0000 7fb9027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.122+0000 7fb9027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.122+0000 7fb9027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.122+0000 7fb9027fc640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:23:20.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.122+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f0004eb0 con 0x7fb90c1430c0 2024-03-25T18:23:20.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.123+0000 7fb9027fc640 10 client.?.objecter ms_handle_connect 0x7fb90c0a98f0 2024-03-25T18:23:20.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.123+0000 7fb9027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:20.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.128+0000 7fb9027fc640 10 client.?.objecter ms_handle_connect 0x7fb90c13f710 2024-03-25T18:23:20.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.128+0000 7fb9027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:20.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.132+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217082528 0 0) 0x7fb90c0aba90 con 0x7fb90c0a98f0 2024-03-25T18:23:20.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.132+0000 7fb9027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.132+0000 7fb9027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.132+0000 7fb9027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.132+0000 7fb9027fc640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:23:20.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.132+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb90c1daca0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.143+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112659700 0 0) 0x7fb90c138ab0 con 0x7fb90c13f710 2024-03-25T18:23:20.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.143+0000 7fb9027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.143+0000 7fb9027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.143+0000 7fb9027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.143+0000 7fb9027fc640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:23:20.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.143+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb90c0aba90 con 0x7fb90c13f710 2024-03-25T18:23:20.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.151+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3063127198 0 0) 0x7fb90c1daca0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.151+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb90c138ab0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.151+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb908001930 con 0x7fb90c0a98f0 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.151+0000 7fb9027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.151+0000 7fb9027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.151+0000 7fb9027fc640 10 monclient(hunting): dump: 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.152+0000 7fb9027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.152+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 931712567 0 0) 0x7fb8f0004eb0 con 0x7fb90c1430c0 2024-03-25T18:23:20.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.152+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb90c1daca0 con 0x7fb90c1430c0 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.152+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb904001f40 con 0x7fb90c1430c0 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.152+0000 7fb9027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.152+0000 7fb9027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.152+0000 7fb9027fc640 10 monclient(hunting): dump: 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:20.155 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.161+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35895499 0 0) 0x7fb90c0aba90 con 0x7fb90c13f710 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.161+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8f0004eb0 con 0x7fb90c13f710 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.161+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb8fc001830 con 0x7fb90c13f710 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.161+0000 7fb9027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.161+0000 7fb9027fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.161+0000 7fb9027fc640 10 monclient(hunting): dump: 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:20.163 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:20.164 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:20.164 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:20.164 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2910558022 0 0) 0x7fb90c138ab0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 >> v1:172.21.15.33:6789/0 conn(0x7fb90c13f710 legacy=0x7fb90c1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 >> v1:172.21.15.22:6789/0 conn(0x7fb90c1430c0 legacy=0x7fb90c1d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 1 monclient: found mon.c 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c1d41e0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb9027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 10 monclient: _renew_subs 2024-03-25T18:23:20.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c1db840 con 0x7fb90c0a98f0 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 10 monclient: _renew_subs 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c138f00 con 0x7fb90c0a98f0 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.171+0000 7fb914084640 1 librados: init done 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.172+0000 7fb914084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:20.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.172+0000 7fb914084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:20.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.172+0000 7fb914084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:20.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.172+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c0627b0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.482+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb908001d40 con 0x7fb90c0a98f0 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.482+0000 7fb9027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.482+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb908004980 con 0x7fb90c0a98f0 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.482+0000 7fb9027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.482+0000 7fb9027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.482+0000 7fb9027fc640 10 monclient: dump: 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.482+0000 7fb903fff640 4 set_mon_vals no callback set 2024-03-25T18:23:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.483+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fb908004ba0 con 0x7fb90c0a98f0 2024-03-25T18:23:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.483+0000 7fb91161f640 10 monclient: get_auth_request con 0x7fb8f006e0f0 auth_method 0 2024-03-25T18:23:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.484+0000 7fb9027fc640 10 client.4409.objecter ms_handle_connect 0x7fb8f006e0f0 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fb908054730 con 0x7fb90c0a98f0 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 10 client.4409.objecter ms_dispatch 0x7fb90c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb908050070 con 0x7fb90c0a98f0 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:20.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.489+0000 7fb9027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:20.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.758+0000 7fb914084640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.758+0000 7fb914084640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.758+0000 7fb914084640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:20.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.758+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb90c1d4370 con 0x7fb90c0a98f0 2024-03-25T18:23:20.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.805+0000 7fb9027fc640 1 -- 172.21.15.22:0/1651168043 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fb908052170 con 0x7fb90c0a98f0 2024-03-25T18:23:20.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.805+0000 7fb9027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.805+0000 7fb9027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:20.807 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:20.807 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:20.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 10 librados: watch_flush enter 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 10 librados: watch_flush exit 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 10 client.4409.objecter successfully canceled tick 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 >> v1:172.21.15.33:6800/215350958 conn(0x7fb8f006e0f0 legacy=0x7fb8f00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 10 monclient: shutdown 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 >> v1:172.21.15.22:6790/0 conn(0x7fb90c0a98f0 legacy=0x7fb90c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 shutdown_connections 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 >> 172.21.15.22:0/1651168043 conn(0x7fb90c0b7ea0 msgr2=0x7fb90c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 shutdown_connections 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 1 -- 172.21.15.22:0/1651168043 wait complete. 2024-03-25T18:23:20.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:20.808+0000 7fb914084640 1 librados: shutdown 2024-03-25T18:23:20.821 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:27.627 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.739+0000 7fa85c088640 10 monclient: get_monmap_and_config 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.739+0000 7fa85c088640 10 monclient: build_initial_monmap 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.739+0000 7fa85c088640 10 monclient: monmap: 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:27.740915+0000 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:27.740915+0000 2024-03-25T18:23:27.741 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:27.742 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:27.742 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:27.742 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:27.742 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:27.742 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:27.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.740+0000 7fa85c088640 1 Processor -- start 2024-03-25T18:23:27.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.740+0000 7fa85c088640 1 -- start start 2024-03-25T18:23:27.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.740+0000 7fa85c088640 10 monclient: init 2024-03-25T18:23:27.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:27.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:27.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 10 monclient(hunting): picked mon.noname-a con 0x7fa8540a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:27.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 10 monclient(hunting): picked mon.noname-c con 0x7fa85413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:27.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 10 monclient(hunting): picked mon.noname-b con 0x7fa8541430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:27.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa854142520 con 0x7fa8540a98f0 2024-03-25T18:23:27.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8540b28a0 con 0x7fa85413f710 2024-03-25T18:23:27.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8540b2a30 con 0x7fa8541430c0 2024-03-25T18:23:27.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:27.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.741+0000 7fa85c088640 10 monclient(hunting): authenticate will time out at 7406.965254s 2024-03-25T18:23:27.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.746+0000 7fa859e24640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa8540a98f0 0x7fa8540a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58532/0 (socket says 172.21.15.22:58532) 2024-03-25T18:23:27.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.746+0000 7fa859e24640 1 -- 172.21.15.22:0/395658678 learned_addr learned my addr 172.21.15.22:0/395658678 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:27.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.752+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257492608 0 0) 0x7fa854142520 con 0x7fa8540a98f0 2024-03-25T18:23:27.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.752+0000 7fa858e22640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.752+0000 7fa858e22640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.752+0000 7fa858e22640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.752+0000 7fa858e22640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:23:27.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.753+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa840003520 con 0x7fa8540a98f0 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715396885 0 0) 0x7fa840003520 con 0x7fa8540a98f0 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 >> v1:172.21.15.22:6790/0 conn(0x7fa8541430c0 legacy=0x7fa85414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 >> v1:172.21.15.33:6789/0 conn(0x7fa85413f710 legacy=0x7fa8541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 10 monclient: _finish_hunting 0 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 1 monclient: found mon.noname-a 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8540b44c0 con 0x7fa8540a98f0 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 10 monclient: _finish_auth 0 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 10 monclient: _check_auth_tickets 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa850001690 con 0x7fa8540a98f0 2024-03-25T18:23:27.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa85c088640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa85c088640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa858e22640 10 monclient: dump: 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:27.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.758+0000 7fa85c088640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa850001860 con 0x7fa8540a98f0 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa858e22640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa858e22640 1 -- 172.21.15.22:0/395658678 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa850001cb0 con 0x7fa8540a98f0 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa85c088640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa85c088640 4 set_mon_vals no callback set 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa858e22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa858e22640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:27.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.781+0000 7fa858e22640 10 monclient: dump: 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:27.783 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:27.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.782+0000 7fa85c088640 10 monclient: shutdown 2024-03-25T18:23:27.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.782+0000 7fa85c088640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.782+0000 7fa85c088640 1 -- 172.21.15.22:0/395658678 >> v1:172.21.15.22:6789/0 conn(0x7fa8540a98f0 legacy=0x7fa8540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.782+0000 7fa85c088640 1 -- 172.21.15.22:0/395658678 shutdown_connections 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.782+0000 7fa85c088640 1 -- 172.21.15.22:0/395658678 >> 172.21.15.22:0/395658678 conn(0x7fa8540b7ea0 msgr2=0x7fa8540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.782+0000 7fa85c088640 1 -- 172.21.15.22:0/395658678 shutdown_connections 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.782+0000 7fa85c088640 1 -- 172.21.15.22:0/395658678 wait complete. 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 10 monclient: build_initial_monmap 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 10 monclient: monmap: 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:27.785 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 1 librados: starting msgr at 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 1 librados: starting objecter 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 1 Processor -- start 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 1 -- start start 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 1 librados: setting wanted keys 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 1 librados: calling monclient init 2024-03-25T18:23:27.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.783+0000 7fa85c088640 10 monclient: init 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 10 monclient(hunting): picked mon.noname-c con 0x7fa8540a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 10 monclient(hunting): picked mon.noname-b con 0x7fa85413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 10 monclient(hunting): picked mon.noname-a con 0x7fa8541430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8540abd30 con 0x7fa8541430c0 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8541d7730 con 0x7fa85413f710 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8541d8360 con 0x7fa8540a98f0 2024-03-25T18:23:27.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:27.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.784+0000 7fa85c088640 10 monclient(hunting): authenticate will time out at 7407.008104s 2024-03-25T18:23:27.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.804+0000 7fa859e24640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa8540a98f0 0x7fa8541d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53340/0 (socket says 172.21.15.22:53340) 2024-03-25T18:23:27.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.804+0000 7fa859e24640 1 -- 172.21.15.22:0/673016401 learned_addr learned my addr 172.21.15.22:0/673016401 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:27.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.805+0000 7fa84a7fc640 10 client.?.objecter ms_handle_connect 0x7fa8540a98f0 2024-03-25T18:23:27.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.805+0000 7fa84a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.807+0000 7fa84a7fc640 10 client.?.objecter ms_handle_connect 0x7fa85413f710 2024-03-25T18:23:27.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.807+0000 7fa84a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.812+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126016946 0 0) 0x7fa8541d8360 con 0x7fa8540a98f0 2024-03-25T18:23:27.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.812+0000 7fa84a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.812+0000 7fa84a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.812+0000 7fa84a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.812+0000 7fa84a7fc640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:23:27.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.812+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa838004eb0 con 0x7fa8540a98f0 2024-03-25T18:23:27.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.815+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588139929 0 0) 0x7fa8541d7730 con 0x7fa85413f710 2024-03-25T18:23:27.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.815+0000 7fa84a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.815+0000 7fa84a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.815+0000 7fa84a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.815+0000 7fa84a7fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:23:27.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.815+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8541d8360 con 0x7fa85413f710 2024-03-25T18:23:27.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.827+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 319845244 0 0) 0x7fa8541d8360 con 0x7fa85413f710 2024-03-25T18:23:27.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.827+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8541d7730 con 0x7fa85413f710 2024-03-25T18:23:27.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.827+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa844001930 con 0x7fa85413f710 2024-03-25T18:23:27.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.827+0000 7fa84a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.827+0000 7fa84a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.827+0000 7fa84a7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:27.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.827+0000 7fa84a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:27.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.834+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 366630895 0 0) 0x7fa8541d7730 con 0x7fa85413f710 2024-03-25T18:23:27.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 >> v1:172.21.15.22:6790/0 conn(0x7fa8540a98f0 legacy=0x7fa8541d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 >> v1:172.21.15.22:6789/0 conn(0x7fa8541430c0 legacy=0x7fa8541d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 1 monclient: found mon.b 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8541380b0 con 0x7fa85413f710 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa84a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 10 monclient: _renew_subs 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8541d5000 con 0x7fa85413f710 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 10 monclient: _renew_subs 2024-03-25T18:23:27.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:27.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8541d43c0 con 0x7fa85413f710 2024-03-25T18:23:27.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.835+0000 7fa85c088640 1 librados: init done 2024-03-25T18:23:27.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.836+0000 7fa85c088640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.836+0000 7fa85c088640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.836+0000 7fa85c088640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:27.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.836+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8541d7590 con 0x7fa85413f710 2024-03-25T18:23:27.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.861+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa844003610 con 0x7fa85413f710 2024-03-25T18:23:27.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.861+0000 7fa84a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:27.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.861+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fa8440037c0 con 0x7fa85413f710 2024-03-25T18:23:27.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.861+0000 7fa84a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.861+0000 7fa84a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:27.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.861+0000 7fa84a7fc640 10 monclient: dump: 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:27.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.861+0000 7fa84bfff640 4 set_mon_vals no callback set 2024-03-25T18:23:27.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.862+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fa8440039e0 con 0x7fa85413f710 2024-03-25T18:23:27.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.862+0000 7fa859e24640 10 monclient: get_auth_request con 0x7fa83806dd70 auth_method 0 2024-03-25T18:23:27.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:27.862+0000 7fa84a7fc640 10 client.4411.objecter ms_handle_connect 0x7fa83806dd70 2024-03-25T18:23:28.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.139+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fa844053450 con 0x7fa85413f710 2024-03-25T18:23:28.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.139+0000 7fa84a7fc640 10 client.4411.objecter ms_dispatch 0x7fa854000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:28.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.139+0000 7fa84a7fc640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:28.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.139+0000 7fa84a7fc640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:28.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.139+0000 7fa84a7fc640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T18:23:28.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.140+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa84404ef60 con 0x7fa85413f710 2024-03-25T18:23:28.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.140+0000 7fa84a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:28.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.140+0000 7fa84a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:28.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.367+0000 7fa85c088640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:28.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.367+0000 7fa85c088640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:28.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.367+0000 7fa85c088640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:28.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.367+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa854138240 con 0x7fa85413f710 2024-03-25T18:23:28.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.397+0000 7fa84a7fc640 1 -- 172.21.15.22:0/673016401 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fa844050e90 con 0x7fa85413f710 2024-03-25T18:23:28.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.397+0000 7fa84a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:28.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.397+0000 7fa84a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:28.399 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:28.399 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 10 librados: watch_flush enter 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 10 librados: watch_flush exit 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 10 client.4411.objecter successfully canceled tick 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 >> v1:172.21.15.33:6800/215350958 conn(0x7fa83806dd70 legacy=0x7fa838070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 10 monclient: shutdown 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 >> v1:172.21.15.33:6789/0 conn(0x7fa85413f710 legacy=0x7fa8541379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 shutdown_connections 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 >> 172.21.15.22:0/673016401 conn(0x7fa8540b7ea0 msgr2=0x7fa854074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 shutdown_connections 2024-03-25T18:23:28.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 1 -- 172.21.15.22:0/673016401 wait complete. 2024-03-25T18:23:28.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:28.400+0000 7fa85c088640 1 librados: shutdown 2024-03-25T18:23:28.413 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:32.981 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:04:35.643 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:32.995 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:04:35.657 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.024 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:04:35.686 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.045 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:04:35.663 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.049 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:04:35.667 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.054 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:04:35.716 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.063 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:04:35.725 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.088 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:04:35.706 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:33.107 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:04:35.769 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:35.219 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.333+0000 7f4f9240f640 10 monclient: get_monmap_and_config 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.333+0000 7f4f9240f640 10 monclient: build_initial_monmap 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.333+0000 7f4f9240f640 10 monclient: monmap: 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:35.334729+0000 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:35.334729+0000 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:35.335 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:35.336 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:35.336 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 1 Processor -- start 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 1 -- start start 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient: init 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c142520 con 0x7f4f8c1430c0 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0b28a0 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0b2a30 con 0x7f4f8c13f710 2024-03-25T18:23:35.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:35.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.334+0000 7f4f9240f640 10 monclient(hunting): authenticate will time out at 7414.558528s 2024-03-25T18:23:35.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.342+0000 7f4f909ac640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f4f8c1430c0 0x7f4f8c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:41176/0 (socket says 172.21.15.22:41176) 2024-03-25T18:23:35.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.342+0000 7f4f909ac640 1 -- 172.21.15.22:0/2050651988 learned_addr learned my addr 172.21.15.22:0/2050651988 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:35.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.354+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137388337 0 0) 0x7f4f8c0b28a0 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.354+0000 7f4f8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.354+0000 7f4f8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.354+0000 7f4f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.354+0000 7f4f8affd640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:23:35.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.354+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f74003520 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.360+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774979306 0 0) 0x7f4f8c142520 con 0x7f4f8c1430c0 2024-03-25T18:23:35.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.360+0000 7f4f8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.360+0000 7f4f8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.360+0000 7f4f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.360+0000 7f4f8affd640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:23:35.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.360+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f8c0b28a0 con 0x7f4f8c1430c0 2024-03-25T18:23:35.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.366+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232352661 0 0) 0x7f4f8c0b2a30 con 0x7f4f8c13f710 2024-03-25T18:23:35.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.366+0000 7f4f8affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.366+0000 7f4f8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.366+0000 7f4f8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.366+0000 7f4f8affd640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:23:35.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.366+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f8c142520 con 0x7f4f8c13f710 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3194343822 0 0) 0x7f4f74003520 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 >> v1:172.21.15.22:6790/0 conn(0x7f4f8c13f710 legacy=0x7f4f8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 >> v1:172.21.15.22:6789/0 conn(0x7f4f8c1430c0 legacy=0x7f4f8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 1 monclient: found mon.noname-c 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0b44c0 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 10 monclient: _finish_auth 0 2024-03-25T18:23:35.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f7c001690 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f8affd640 10 monclient: dump: 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:35.374 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:35.375 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:35.375 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:35.375 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:35.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f9240f640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:23:35.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.371+0000 7f4f9240f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:35.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f7c0017f0 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:35.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f8affd640 1 -- 172.21.15.22:0/2050651988 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f7c001c00 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f9240f640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:35.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f9240f640 4 set_mon_vals no callback set 2024-03-25T18:23:35.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.389+0000 7f4f8affd640 10 monclient: dump: 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:35.391 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:35.392 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:35.392 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:35.392 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:35.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.390+0000 7f4f9240f640 10 monclient: shutdown 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.390+0000 7f4f9240f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.390+0000 7f4f9240f640 1 -- 172.21.15.22:0/2050651988 >> v1:172.21.15.33:6789/0 conn(0x7f4f8c0a98f0 legacy=0x7f4f8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.390+0000 7f4f9240f640 1 -- 172.21.15.22:0/2050651988 shutdown_connections 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.390+0000 7f4f9240f640 1 -- 172.21.15.22:0/2050651988 >> 172.21.15.22:0/2050651988 conn(0x7f4f8c0b7ea0 msgr2=0x7f4f8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.390+0000 7f4f9240f640 1 -- 172.21.15.22:0/2050651988 shutdown_connections 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.390+0000 7f4f9240f640 1 -- 172.21.15.22:0/2050651988 wait complete. 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 10 monclient: build_initial_monmap 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 10 monclient: monmap: 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:35.395 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 1 librados: starting msgr at 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 1 librados: starting objecter 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 1 Processor -- start 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 1 -- start start 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 1 librados: setting wanted keys 2024-03-25T18:23:35.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 1 librados: calling monclient init 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.391+0000 7f4f9240f640 10 monclient: init 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f8c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f8c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f8c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c139a40 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c133f20 con 0x7f4f8c1430c0 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0ab950 con 0x7f4f8c13f710 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:35.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.392+0000 7f4f9240f640 10 monclient(hunting): authenticate will time out at 7414.615811s 2024-03-25T18:23:35.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.409+0000 7f4f909ac640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f4f8c1430c0 0x7f4f8c133810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:53902/0 (socket says 172.21.15.22:53902) 2024-03-25T18:23:35.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.409+0000 7f4f909ac640 1 -- 172.21.15.22:0/3942370777 learned_addr learned my addr 172.21.15.22:0/3942370777 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:35.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.410+0000 7f4f88ff9640 10 client.?.objecter ms_handle_connect 0x7f4f8c1430c0 2024-03-25T18:23:35.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.410+0000 7f4f88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:35.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.412+0000 7f4f88ff9640 10 client.?.objecter ms_handle_connect 0x7f4f8c0a98f0 2024-03-25T18:23:35.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.412+0000 7f4f88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:35.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.414+0000 7f4f88ff9640 10 client.?.objecter ms_handle_connect 0x7f4f8c13f710 2024-03-25T18:23:35.416 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.414+0000 7f4f88ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:35.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.418+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425818794 0 0) 0x7f4f8c133f20 con 0x7f4f8c1430c0 2024-03-25T18:23:35.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.419+0000 7f4f88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.419+0000 7f4f88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.419+0000 7f4f88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.419+0000 7f4f88ff9640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:23:35.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.419+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78004eb0 con 0x7f4f8c1430c0 2024-03-25T18:23:35.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.424+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220481680 0 0) 0x7f4f8c139a40 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.424+0000 7f4f88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.424+0000 7f4f88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.424+0000 7f4f88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.424+0000 7f4f88ff9640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:23:35.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.424+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f8c133f20 con 0x7f4f8c0a98f0 2024-03-25T18:23:35.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.427+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809129160 0 0) 0x7f4f78004eb0 con 0x7f4f8c1430c0 2024-03-25T18:23:35.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.427+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f8c139a40 con 0x7f4f8c1430c0 2024-03-25T18:23:35.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.427+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f84001c70 con 0x7f4f8c1430c0 2024-03-25T18:23:35.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.427+0000 7f4f88ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.427+0000 7f4f88ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:35.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.427+0000 7f4f88ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:35.429 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:35.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.427+0000 7f4f88ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:35.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.432+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858059907 0 0) 0x7f4f8c0ab950 con 0x7f4f8c13f710 2024-03-25T18:23:35.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.432+0000 7f4f88ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.432+0000 7f4f88ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.432+0000 7f4f88ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.432+0000 7f4f88ff9640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:23:35.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.432+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78004eb0 con 0x7f4f8c13f710 2024-03-25T18:23:35.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1092079199 0 0) 0x7f4f8c139a40 con 0x7f4f8c1430c0 2024-03-25T18:23:35.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 >> v1:172.21.15.22:6790/0 conn(0x7f4f8c13f710 legacy=0x7f4f8c133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 >> v1:172.21.15.22:6789/0 conn(0x7f4f8c0a98f0 legacy=0x7f4f8c138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 1 monclient: found mon.b 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0abc30 con 0x7f4f8c1430c0 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 10 monclient: _renew_subs 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c0abf10 con 0x7f4f8c1430c0 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 10 monclient: _renew_subs 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:35.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c139370 con 0x7f4f8c1430c0 2024-03-25T18:23:35.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.442+0000 7f4f9240f640 1 librados: init done 2024-03-25T18:23:35.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.443+0000 7f4f9240f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:35.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.443+0000 7f4f9240f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:35.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.443+0000 7f4f9240f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:35.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.443+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8c0627b0 con 0x7f4f8c1430c0 2024-03-25T18:23:35.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.740+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f84004b60 con 0x7f4f8c1430c0 2024-03-25T18:23:35.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.740+0000 7f4f88ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:35.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.740+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4f84004e70 con 0x7f4f8c1430c0 2024-03-25T18:23:35.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.740+0000 7f4f88ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.740+0000 7f4f88ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.740+0000 7f4f88ff9640 10 monclient: dump: 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:35.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.740+0000 7f4f8a7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:35.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.741+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f4f8401ae40 con 0x7f4f8c1430c0 2024-03-25T18:23:35.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.742+0000 7f4f8bfff640 10 monclient: get_auth_request con 0x7f4f7806de00 auth_method 0 2024-03-25T18:23:35.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.742+0000 7f4f88ff9640 10 client.4417.objecter ms_handle_connect 0x7f4f7806de00 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.745+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f4f84054b00 con 0x7f4f8c1430c0 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.745+0000 7f4f88ff9640 10 client.4417.objecter ms_dispatch 0x7f4f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.745+0000 7f4f88ff9640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.745+0000 7f4f88ff9640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.745+0000 7f4f88ff9640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.746+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f84050440 con 0x7f4f8c1430c0 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.746+0000 7f4f88ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:35.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:35.746+0000 7f4f88ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:36.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.009+0000 7f4f9240f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:36.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.009+0000 7f4f9240f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:36.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.009+0000 7f4f9240f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:36.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.009+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f8c0b7180 con 0x7f4f8c1430c0 2024-03-25T18:23:36.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.037+0000 7f4f88ff9640 1 -- 172.21.15.22:0/3942370777 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f4f84052540 con 0x7f4f8c1430c0 2024-03-25T18:23:36.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.037+0000 7f4f88ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:36.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.037+0000 7f4f88ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:36.039 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:36.039 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 10 librados: watch_flush enter 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 10 librados: watch_flush exit 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 10 client.4417.objecter successfully canceled tick 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 >> v1:172.21.15.33:6800/215350958 conn(0x7f4f7806de00 legacy=0x7f4f780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 10 monclient: shutdown 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 >> v1:172.21.15.33:6789/0 conn(0x7f4f8c1430c0 legacy=0x7f4f8c133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 shutdown_connections 2024-03-25T18:23:36.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 >> 172.21.15.22:0/3942370777 conn(0x7f4f8c0b7ea0 msgr2=0x7f4f8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:36.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 shutdown_connections 2024-03-25T18:23:36.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 1 -- 172.21.15.22:0/3942370777 wait complete. 2024-03-25T18:23:36.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:36.039+0000 7f4f9240f640 1 librados: shutdown 2024-03-25T18:23:36.052 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:42.857 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:42.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.971+0000 7f87dc790640 10 monclient: get_monmap_and_config 2024-03-25T18:23:42.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.971+0000 7f87dc790640 10 monclient: build_initial_monmap 2024-03-25T18:23:42.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.971+0000 7f87dc790640 10 monclient: monmap: 2024-03-25T18:23:42.972 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:42.972 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:42.972628+0000 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:42.972628+0000 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:42.973 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.971+0000 7f87dc790640 1 Processor -- start 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 1 -- start start 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 10 monclient: init 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 10 monclient(hunting): picked mon.noname-c con 0x7f87d40a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 10 monclient(hunting): picked mon.noname-a con 0x7f87d413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 10 monclient(hunting): picked mon.noname-b con 0x7f87d41430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d4142520 con 0x7f87d413f710 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.972+0000 7f87dc790640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d40b28a0 con 0x7f87d40a98f0 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.973+0000 7f87dc790640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d40b2a30 con 0x7f87d41430c0 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.973+0000 7f87dc790640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:42.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.973+0000 7f87dc790640 10 monclient(hunting): authenticate will time out at 7422.196589s 2024-03-25T18:23:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.976+0000 7f87da52c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f87d40a98f0 0x7f87d40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:58424/0 (socket says 172.21.15.22:58424) 2024-03-25T18:23:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.976+0000 7f87da52c640 1 -- 172.21.15.22:0/2865621963 learned_addr learned my addr 172.21.15.22:0/2865621963 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:42.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.984+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124013235 0 0) 0x7f87d40b28a0 con 0x7f87d40a98f0 2024-03-25T18:23:42.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.984+0000 7f87d952a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:42.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.984+0000 7f87d952a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:42.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.984+0000 7f87d952a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:42.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.984+0000 7f87d952a640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:23:42.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.984+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87c0003520 con 0x7f87d40a98f0 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2751369494 0 0) 0x7f87c0003520 con 0x7f87d40a98f0 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 >> v1:172.21.15.22:6790/0 conn(0x7f87d41430c0 legacy=0x7f87d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 >> v1:172.21.15.22:6789/0 conn(0x7f87d413f710 legacy=0x7f87d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 10 monclient: _finish_hunting 0 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 1 monclient: found mon.noname-c 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d40b44c0 con 0x7f87d40a98f0 2024-03-25T18:23:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 10 monclient: _finish_auth 0 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 10 monclient: _check_auth_tickets 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f87d0001690 con 0x7f87d40a98f0 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87d952a640 10 monclient: dump: 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:42.997 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:42.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87dc790640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:23:42.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:42.994+0000 7f87dc790640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87d00017f0 con 0x7f87d40a98f0 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87d952a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87d952a640 1 -- 172.21.15.22:0/2865621963 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f87d0001c00 con 0x7f87d40a98f0 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87d952a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87d952a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87d952a640 10 monclient: dump: 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:43.014 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:43.015 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:43.015 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:43.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87dc790640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:43.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.012+0000 7f87dc790640 4 set_mon_vals no callback set 2024-03-25T18:23:43.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.013+0000 7f87dc790640 10 monclient: shutdown 2024-03-25T18:23:43.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.013+0000 7f87dc790640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:43.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.013+0000 7f87dc790640 1 -- 172.21.15.22:0/2865621963 >> v1:172.21.15.33:6789/0 conn(0x7f87d40a98f0 legacy=0x7f87d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.013+0000 7f87dc790640 1 -- 172.21.15.22:0/2865621963 shutdown_connections 2024-03-25T18:23:43.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.013+0000 7f87dc790640 1 -- 172.21.15.22:0/2865621963 >> 172.21.15.22:0/2865621963 conn(0x7f87d40b7ea0 msgr2=0x7f87d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:43.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.014+0000 7f87dc790640 1 -- 172.21.15.22:0/2865621963 shutdown_connections 2024-03-25T18:23:43.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.014+0000 7f87dc790640 1 -- 172.21.15.22:0/2865621963 wait complete. 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 10 monclient: build_initial_monmap 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 10 monclient: monmap: 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 1 librados: starting msgr at 2024-03-25T18:23:43.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 1 librados: starting objecter 2024-03-25T18:23:43.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 1 Processor -- start 2024-03-25T18:23:43.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 1 -- start start 2024-03-25T18:23:43.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 1 librados: setting wanted keys 2024-03-25T18:23:43.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 1 librados: calling monclient init 2024-03-25T18:23:43.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.015+0000 7f87dc790640 10 monclient: init 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 10 monclient(hunting): picked mon.noname-b con 0x7f87d40a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 10 monclient(hunting): picked mon.noname-a con 0x7f87d413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 10 monclient(hunting): picked mon.noname-c con 0x7f87d41430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d41df1f0 con 0x7f87d413f710 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d41dfe20 con 0x7f87d40a98f0 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d41dffb0 con 0x7f87d41430c0 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:43.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.016+0000 7f87dc790640 10 monclient(hunting): authenticate will time out at 7422.240311s 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.029+0000 7f87da52c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f87d40a98f0 0x7f87d41df710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:58470/0 (socket says 172.21.15.22:58470) 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.029+0000 7f87da52c640 1 -- 172.21.15.22:0/2379726821 learned_addr learned my addr 172.21.15.22:0/2379726821 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:43.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.033+0000 7f87caffd640 10 client.?.objecter ms_handle_connect 0x7f87d41430c0 2024-03-25T18:23:43.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.033+0000 7f87caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:43.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.034+0000 7f87caffd640 10 client.?.objecter ms_handle_connect 0x7f87d40a98f0 2024-03-25T18:23:43.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.034+0000 7f87caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:43.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.035+0000 7f87caffd640 10 client.?.objecter ms_handle_connect 0x7f87d413f710 2024-03-25T18:23:43.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.035+0000 7f87caffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:43.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.038+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861956198 0 0) 0x7f87d40b2a30 con 0x7f87d41430c0 2024-03-25T18:23:43.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.038+0000 7f87caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.038+0000 7f87caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.038+0000 7f87caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.038+0000 7f87caffd640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:23:43.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.038+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8004eb0 con 0x7f87d41430c0 2024-03-25T18:23:43.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.042+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442420713 0 0) 0x7f87d41dfe20 con 0x7f87d40a98f0 2024-03-25T18:23:43.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.042+0000 7f87caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.042+0000 7f87caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.042+0000 7f87caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.042+0000 7f87caffd640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:23:43.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.042+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87d40b2a30 con 0x7f87d40a98f0 2024-03-25T18:23:43.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.048+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2506333892 0 0) 0x7f87b8004eb0 con 0x7f87d41430c0 2024-03-25T18:23:43.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.048+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d41dfe20 con 0x7f87d41430c0 2024-03-25T18:23:43.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.048+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f87cc0019f0 con 0x7f87d41430c0 2024-03-25T18:23:43.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): dump: 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83196445 0 0) 0x7f87d41df1f0 con 0x7f87d413f710 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:23:43.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.049+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87b8004eb0 con 0x7f87d413f710 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.053+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2742816749 0 0) 0x7f87d40b2a30 con 0x7f87d40a98f0 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.054+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d41df1f0 con 0x7f87d40a98f0 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.054+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f87d0001b40 con 0x7f87d40a98f0 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.054+0000 7f87caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.054+0000 7f87caffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.054+0000 7f87caffd640 10 monclient(hunting): dump: 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.055 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:43.056 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:43.056 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:43.056 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:43.056 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:43.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.059+0000 7f87d9d2b640 1 -- 172.21.15.22:0/2379726821 >> v1:172.21.15.22:6789/0 conn(0x7f87d413f710 legacy=0x7f87d406efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:23:43.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.059+0000 7f87d9d2b640 1 -- 172.21.15.22:0/2379726821 >> v1:172.21.15.22:6789/0 conn(0x7f87d413f710 legacy=0x7f87d406efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:23:43.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.059+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988333427 0 0) 0x7f87b8004eb0 con 0x7f87d413f710 2024-03-25T18:23:43.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.059+0000 7f87d9d2b640 1 --1- 172.21.15.22:0/2379726821 >> v1:172.21.15.22:6789/0 conn(0x7f87d413f710 0x7f87d406efa0 :-1 s=OPENED pgs=28914 cs=1 l=1).handle_message read tag failed 2024-03-25T18:23:43.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.059+0000 7f87d9d2b640 1 --1- 172.21.15.22:0/2379726821 >> v1:172.21.15.22:6789/0 conn(0x7f87d413f710 0x7f87d406efa0 :-1 s=OPENED pgs=28914 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:23:43.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.059+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87d40b2a30 con 0x7f87d413f710 2024-03-25T18:23:43.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.059+0000 7f87caffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.22:6789/0 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 936461598 0 0) 0x7f87d41dfe20 con 0x7f87d41430c0 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 >> v1:172.21.15.33:6789/0 conn(0x7f87d40a98f0 legacy=0x7f87d41df710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 >> v1:172.21.15.22:6789/0 conn(0x7f87d413f710 legacy=0x7f87d406efa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 1 monclient: found mon.c 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:43.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d406f6b0 con 0x7f87d41430c0 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 10 monclient: _renew_subs 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d4070c80 con 0x7f87d41430c0 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 10 monclient: _renew_subs 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d406e200 con 0x7f87d41430c0 2024-03-25T18:23:43.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.064+0000 7f87dc790640 1 librados: init done 2024-03-25T18:23:43.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.066+0000 7f87dc790640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:43.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.066+0000 7f87dc790640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:43.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.066+0000 7f87dc790640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:43.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.066+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d40627b0 con 0x7f87d41430c0 2024-03-25T18:23:43.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.387+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f87cc001ee0 con 0x7f87d41430c0 2024-03-25T18:23:43.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.387+0000 7f87caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.387+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f87cc003b10 con 0x7f87d41430c0 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.387+0000 7f87caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.387+0000 7f87caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.387+0000 7f87caffd640 10 monclient: dump: 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:43.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.387+0000 7f87d8d29640 4 set_mon_vals no callback set 2024-03-25T18:23:43.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.388+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f87cc003d30 con 0x7f87d41430c0 2024-03-25T18:23:43.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.388+0000 7f87da52c640 10 monclient: get_auth_request con 0x7f87b806e170 auth_method 0 2024-03-25T18:23:43.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.388+0000 7f87caffd640 10 client.4424.objecter ms_handle_connect 0x7f87b806e170 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f87cc053650 con 0x7f87d41430c0 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 10 client.4424.objecter ms_dispatch 0x7f87d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87cc04f160 con 0x7f87d41430c0 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:43.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.396+0000 7f87caffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:43.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.639+0000 7f87dc790640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.639+0000 7f87dc790640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.639+0000 7f87dc790640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:43.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.639+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87d406f840 con 0x7f87d41430c0 2024-03-25T18:23:43.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.716+0000 7f87caffd640 1 -- 172.21.15.22:0/2379726821 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f87cc051090 con 0x7f87d41430c0 2024-03-25T18:23:43.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.716+0000 7f87caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.716+0000 7f87caffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:43.717 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:43.718 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:43.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 10 librados: watch_flush enter 2024-03-25T18:23:43.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 10 librados: watch_flush exit 2024-03-25T18:23:43.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T18:23:43.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 10 client.4424.objecter successfully canceled tick 2024-03-25T18:23:43.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 >> v1:172.21.15.33:6800/215350958 conn(0x7f87b806e170 legacy=0x7f87b8070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 10 monclient: shutdown 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 >> v1:172.21.15.22:6790/0 conn(0x7f87d41430c0 legacy=0x7f87d41e65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 shutdown_connections 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 >> 172.21.15.22:0/2379726821 conn(0x7f87d40b7ea0 msgr2=0x7f87d414e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 shutdown_connections 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 1 -- 172.21.15.22:0/2379726821 wait complete. 2024-03-25T18:23:43.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:43.718+0000 7f87dc790640 1 librados: shutdown 2024-03-25T18:23:43.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:50.537 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:50.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.649+0000 7feb0fcd3640 10 monclient: get_monmap_and_config 2024-03-25T18:23:50.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.649+0000 7feb0fcd3640 10 monclient: build_initial_monmap 2024-03-25T18:23:50.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.649+0000 7feb0fcd3640 10 monclient: monmap: 2024-03-25T18:23:50.649 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:50.649690+0000 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:50.649690+0000 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:50.650 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.650+0000 7feb0fcd3640 1 Processor -- start 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.650+0000 7feb0fcd3640 1 -- start start 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.650+0000 7feb0fcd3640 10 monclient: init 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 10 monclient(hunting): picked mon.noname-a con 0x7feb080a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 10 monclient(hunting): picked mon.noname-c con 0x7feb0813f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 10 monclient(hunting): picked mon.noname-b con 0x7feb081430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb08142520 con 0x7feb080a98f0 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb080b28a0 con 0x7feb0813f710 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb080b2a30 con 0x7feb081430c0 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.651+0000 7feb0fcd3640 10 monclient(hunting): authenticate will time out at 7429.873938s 2024-03-25T18:23:50.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.664+0000 7feb0da6f640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7feb080a98f0 0x7feb080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:45270/0 (socket says 172.21.15.22:45270) 2024-03-25T18:23:50.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.664+0000 7feb0da6f640 1 -- 172.21.15.22:0/3795644099 learned_addr learned my addr 172.21.15.22:0/3795644099 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:50.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.677+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 793699350 0 0) 0x7feb080b28a0 con 0x7feb0813f710 2024-03-25T18:23:50.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.678+0000 7feb0ca6d640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.678+0000 7feb0ca6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.678+0000 7feb0ca6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.678+0000 7feb0ca6d640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:23:50.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.678+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf4003520 con 0x7feb0813f710 2024-03-25T18:23:50.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.681+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355777181 0 0) 0x7feb08142520 con 0x7feb080a98f0 2024-03-25T18:23:50.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.681+0000 7feb0ca6d640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.681+0000 7feb0ca6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.681+0000 7feb0ca6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.681+0000 7feb0ca6d640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:23:50.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.681+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb080b28a0 con 0x7feb080a98f0 2024-03-25T18:23:50.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.694+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633602782 0 0) 0x7feb080b2a30 con 0x7feb081430c0 2024-03-25T18:23:50.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.694+0000 7feb0ca6d640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.694+0000 7feb0ca6d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.694+0000 7feb0ca6d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.694+0000 7feb0ca6d640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:23:50.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.694+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb08142520 con 0x7feb081430c0 2024-03-25T18:23:50.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.710+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4118644465 0 0) 0x7feaf4003520 con 0x7feb0813f710 2024-03-25T18:23:50.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 >> v1:172.21.15.22:6790/0 conn(0x7feb081430c0 legacy=0x7feb0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 >> v1:172.21.15.22:6789/0 conn(0x7feb080a98f0 legacy=0x7feb080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 10 monclient: _finish_hunting 0 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 1 monclient: found mon.noname-c 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080b44c0 con 0x7feb0813f710 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 10 monclient: _finish_auth 0 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 10 monclient: _check_auth_tickets 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7feaf8001690 con 0x7feb0813f710 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0ca6d640 10 monclient: dump: 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.712 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0fcd3640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:23:50.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.711+0000 7feb0fcd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:50.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.727+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaf80017f0 con 0x7feb0813f710 2024-03-25T18:23:50.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.727+0000 7feb0ca6d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:50.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.727+0000 7feb0ca6d640 1 -- 172.21.15.22:0/3795644099 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7feaf8001c00 con 0x7feb0813f710 2024-03-25T18:23:50.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.727+0000 7feb0fcd3640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:50.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0fcd3640 4 set_mon_vals no callback set 2024-03-25T18:23:50.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0ca6d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0ca6d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0ca6d640 10 monclient: dump: 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:50.729 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0fcd3640 10 monclient: shutdown 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0fcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0fcd3640 1 -- 172.21.15.22:0/3795644099 >> v1:172.21.15.33:6789/0 conn(0x7feb0813f710 legacy=0x7feb081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0fcd3640 1 -- 172.21.15.22:0/3795644099 shutdown_connections 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.728+0000 7feb0fcd3640 1 -- 172.21.15.22:0/3795644099 >> 172.21.15.22:0/3795644099 conn(0x7feb080b7ea0 msgr2=0x7feb080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.729+0000 7feb0fcd3640 1 -- 172.21.15.22:0/3795644099 shutdown_connections 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.729+0000 7feb0fcd3640 1 -- 172.21.15.22:0/3795644099 wait complete. 2024-03-25T18:23:50.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.729+0000 7feb0fcd3640 10 monclient: build_initial_monmap 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.729+0000 7feb0fcd3640 10 monclient: monmap: 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.730+0000 7feb0fcd3640 1 librados: starting msgr at 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.730+0000 7feb0fcd3640 1 librados: starting objecter 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.730+0000 7feb0fcd3640 1 Processor -- start 2024-03-25T18:23:50.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.730+0000 7feb0fcd3640 1 -- start start 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.730+0000 7feb0fcd3640 1 librados: setting wanted keys 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.730+0000 7feb0fcd3640 1 librados: calling monclient init 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.730+0000 7feb0fcd3640 10 monclient: init 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 10 monclient(hunting): picked mon.noname-b con 0x7feb080a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 10 monclient(hunting): picked mon.noname-c con 0x7feb0813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 10 monclient(hunting): picked mon.noname-a con 0x7feb081430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb08139070 con 0x7feb081430c0 2024-03-25T18:23:50.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb08133710 con 0x7feb080a98f0 2024-03-25T18:23:50.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb081338a0 con 0x7feb0813f710 2024-03-25T18:23:50.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:50.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.731+0000 7feb0fcd3640 10 monclient(hunting): authenticate will time out at 7429.953899s 2024-03-25T18:23:50.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.741+0000 7feb0da6f640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7feb080a98f0 0x7feb081378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:57172/0 (socket says 172.21.15.22:57172) 2024-03-25T18:23:50.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.741+0000 7feb0da6f640 1 -- 172.21.15.22:0/2208669248 learned_addr learned my addr 172.21.15.22:0/2208669248 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:50.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.743+0000 7feafe7fc640 10 client.?.objecter ms_handle_connect 0x7feb080a98f0 2024-03-25T18:23:50.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.743+0000 7feafe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.746+0000 7feafe7fc640 10 client.?.objecter ms_handle_connect 0x7feb0813f710 2024-03-25T18:23:50.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.746+0000 7feafe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.752+0000 7feafe7fc640 10 client.?.objecter ms_handle_connect 0x7feb081430c0 2024-03-25T18:23:50.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.752+0000 7feafe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.759+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266920426 0 0) 0x7feb08133710 con 0x7feb080a98f0 2024-03-25T18:23:50.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.759+0000 7feafe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.759+0000 7feafe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.759+0000 7feafe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.759+0000 7feafe7fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.759+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaec004eb0 con 0x7feb080a98f0 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.760+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3221085420 0 0) 0x7feb081338a0 con 0x7feb0813f710 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.760+0000 7feafe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.760+0000 7feafe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.760 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.760+0000 7feafe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.760+0000 7feafe7fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:23:50.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.760+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb08133710 con 0x7feb0813f710 2024-03-25T18:23:50.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.765+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069318745 0 0) 0x7feb08139070 con 0x7feb081430c0 2024-03-25T18:23:50.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.765+0000 7feafe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.765+0000 7feafe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.765+0000 7feafe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.765+0000 7feafe7fc640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:23:50.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.765+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb081338a0 con 0x7feb081430c0 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.778+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482932892 0 0) 0x7feb08133710 con 0x7feb0813f710 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.779+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb08139070 con 0x7feb0813f710 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.779+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7feaf8001b00 con 0x7feb0813f710 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.779+0000 7feafe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.779+0000 7feafe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.779+0000 7feafe7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:50.779 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:50.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.779+0000 7feafe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:50.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.781+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 916427510 0 0) 0x7feb081338a0 con 0x7feb081430c0 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.781+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb08133710 con 0x7feb081430c0 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.781+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7feb00001bd0 con 0x7feb081430c0 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.781+0000 7feafe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.781+0000 7feafe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.781+0000 7feafe7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:50.782 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:50.783 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:50.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.784+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 794483852 0 0) 0x7feaec004eb0 con 0x7feb080a98f0 2024-03-25T18:23:50.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.784+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb081338a0 con 0x7feb080a98f0 2024-03-25T18:23:50.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.784+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7feb040019f0 con 0x7feb080a98f0 2024-03-25T18:23:50.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.784+0000 7feafe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.784+0000 7feafe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:50.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.784+0000 7feafe7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:50.785 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:50.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3580735450 0 0) 0x7feb08139070 con 0x7feb0813f710 2024-03-25T18:23:50.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 >> v1:172.21.15.33:6789/0 conn(0x7feb080a98f0 legacy=0x7feb081378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 >> v1:172.21.15.22:6789/0 conn(0x7feb081430c0 legacy=0x7feb08133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 1 monclient: found mon.c 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb08133a30 con 0x7feb0813f710 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.798+0000 7feafe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feafe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 5 monclient: authenticate success, global_id 4430 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 10 monclient: _renew_subs 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb080b7100 con 0x7feb0813f710 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 10 monclient: _renew_subs 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:50.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb08136e60 con 0x7feb0813f710 2024-03-25T18:23:50.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.799+0000 7feb0fcd3640 1 librados: init done 2024-03-25T18:23:50.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.800+0000 7feb0fcd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:50.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.800+0000 7feb0fcd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:50.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.800+0000 7feb0fcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:50.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:50.800+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb08074cc0 con 0x7feb0813f710 2024-03-25T18:23:51.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.137+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaf8001ca0 con 0x7feb0813f710 2024-03-25T18:23:51.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.137+0000 7feafe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:51.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.137+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7feaf8003a40 con 0x7feb0813f710 2024-03-25T18:23:51.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.137+0000 7feafe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:51.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.137+0000 7feafe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.137+0000 7feafe7fc640 10 monclient: dump: 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:51.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.137+0000 7feafffff640 4 set_mon_vals no callback set 2024-03-25T18:23:51.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.138+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7feaf8019a10 con 0x7feb0813f710 2024-03-25T18:23:51.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.138+0000 7feb0da6f640 10 monclient: get_auth_request con 0x7feaec06e0f0 auth_method 0 2024-03-25T18:23:51.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.138+0000 7feafe7fc640 10 client.4430.objecter ms_handle_connect 0x7feaec06e0f0 2024-03-25T18:23:51.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.143+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7feaf80535b0 con 0x7feb0813f710 2024-03-25T18:23:51.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.143+0000 7feafe7fc640 10 client.4430.objecter ms_dispatch 0x7feb08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:51.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.144+0000 7feafe7fc640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:51.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.144+0000 7feafe7fc640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:51.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.144+0000 7feafe7fc640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-25T18:23:51.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.144+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feaf804f0c0 con 0x7feb0813f710 2024-03-25T18:23:51.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.144+0000 7feafe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:51.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.144+0000 7feafe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:51.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.380+0000 7feb0fcd3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:51.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.380+0000 7feb0fcd3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:51.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.380+0000 7feb0fcd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:23:51.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.380+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb080627b0 con 0x7feb0813f710 2024-03-25T18:23:51.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.414+0000 7feafe7fc640 1 -- 172.21.15.22:0/2208669248 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7feaf8050ff0 con 0x7feb0813f710 2024-03-25T18:23:51.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.414+0000 7feafe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:51.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.414+0000 7feafe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:51.415 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:51.415 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:51.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 10 librados: watch_flush enter 2024-03-25T18:23:51.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 10 librados: watch_flush exit 2024-03-25T18:23:51.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 10 client.4430.objecter successfully canceled tick 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 >> v1:172.21.15.33:6800/215350958 conn(0x7feaec06e0f0 legacy=0x7feaec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 10 monclient: shutdown 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 >> v1:172.21.15.22:6790/0 conn(0x7feb0813f710 legacy=0x7feb081daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 shutdown_connections 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 >> 172.21.15.22:0/2208669248 conn(0x7feb080b7ea0 msgr2=0x7feb08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 shutdown_connections 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 1 -- 172.21.15.22:0/2208669248 wait complete. 2024-03-25T18:23:51.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:51.417+0000 7feb0fcd3640 1 librados: shutdown 2024-03-25T18:23:51.430 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:53.023 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:04:55.685 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.030 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:04:55.692 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.087 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:04:55.749 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.090 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:04:55.709 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.095 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:04:55.713 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.100 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:04:55.762 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.114 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:04:55.776 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.120 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:04:55.738 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:53.144 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:04:55.806 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:58.237 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:58.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.352+0000 7f5d2e4c1640 10 monclient: get_monmap_and_config 2024-03-25T18:23:58.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.352+0000 7f5d2e4c1640 10 monclient: build_initial_monmap 2024-03-25T18:23:58.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.352+0000 7f5d2e4c1640 10 monclient: monmap: 2024-03-25T18:23:58.352 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:58.352 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:58.352 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:23:58.352471+0000 2024-03-25T18:23:58.352 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:23:58.352471+0000 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 1 Processor -- start 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 1 -- start start 2024-03-25T18:23:58.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 10 monclient: init 2024-03-25T18:23:58.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:58.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:58.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 10 monclient(hunting): picked mon.noname-b con 0x7f5d280b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:58.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 10 monclient(hunting): picked mon.noname-a con 0x7f5d280b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:58.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2814be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:58.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.353+0000 7f5d2e4c1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d280af940 con 0x7f5d280b1730 2024-03-25T18:23:58.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.354+0000 7f5d2e4c1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d280b0a70 con 0x7f5d2814be90 2024-03-25T18:23:58.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.354+0000 7f5d2e4c1640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d280b0d50 con 0x7f5d280b4d60 2024-03-25T18:23:58.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.354+0000 7f5d2e4c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:58.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.354+0000 7f5d2e4c1640 10 monclient(hunting): authenticate will time out at 7437.576609s 2024-03-25T18:23:58.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.358+0000 7f5d2ca5e640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f5d2814be90 0x7f5d2814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:56440/0 (socket says 172.21.15.22:56440) 2024-03-25T18:23:58.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.358+0000 7f5d2ca5e640 1 -- 172.21.15.22:0/1446615475 learned_addr learned my addr 172.21.15.22:0/1446615475 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.367+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2272139565 0 0) 0x7f5d280b0a70 con 0x7f5d2814be90 2024-03-25T18:23:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.367+0000 7f5d26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.367+0000 7f5d26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.367+0000 7f5d26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.367+0000 7f5d26ffd640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:23:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.367+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d10003520 con 0x7f5d2814be90 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3722052423 0 0) 0x7f5d10003520 con 0x7f5d2814be90 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 >> v1:172.21.15.22:6790/0 conn(0x7f5d280b4d60 legacy=0x7f5d2813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 >> v1:172.21.15.22:6789/0 conn(0x7f5d280b1730 legacy=0x7f5d280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 1 monclient: found mon.noname-c 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d280b1030 con 0x7f5d2814be90 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:58.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5d20001a10 con 0x7f5d2814be90 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d26ffd640 10 monclient: dump: 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d2e4c1640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:23:58.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.376+0000 7f5d2e4c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d20001b70 con 0x7f5d2814be90 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d26ffd640 1 -- 172.21.15.22:0/1446615475 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5d20001fc0 con 0x7f5d2814be90 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d2e4c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d2e4c1640 4 set_mon_vals no callback set 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.382+0000 7f5d26ffd640 10 monclient: dump: 2024-03-25T18:23:58.382 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:58.383 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.383+0000 7f5d2e4c1640 10 monclient: shutdown 2024-03-25T18:23:58.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.383+0000 7f5d2e4c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:58.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.383+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/1446615475 >> v1:172.21.15.33:6789/0 conn(0x7f5d2814be90 legacy=0x7f5d2814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.383+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/1446615475 shutdown_connections 2024-03-25T18:23:58.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.383+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/1446615475 >> 172.21.15.22:0/1446615475 conn(0x7f5d280617a0 msgr2=0x7f5d28061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.383+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/1446615475 shutdown_connections 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.383+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/1446615475 wait complete. 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 10 monclient: build_initial_monmap 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 10 monclient: monmap: 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 1 librados: starting msgr at 2024-03-25T18:23:58.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 1 librados: starting objecter 2024-03-25T18:23:58.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 1 Processor -- start 2024-03-25T18:23:58.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 1 -- start start 2024-03-25T18:23:58.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 1 librados: setting wanted keys 2024-03-25T18:23:58.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 1 librados: calling monclient init 2024-03-25T18:23:58.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.384+0000 7f5d2e4c1640 10 monclient: init 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 10 monclient(hunting): picked mon.noname-b con 0x7f5d280b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 10 monclient(hunting): picked mon.noname-a con 0x7f5d280b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 10 monclient(hunting): picked mon.noname-c con 0x7f5d2814be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d28140f70 con 0x7f5d280b4d60 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d28143130 con 0x7f5d280b1730 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d28141fa0 con 0x7f5d2814be90 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:58.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.385+0000 7f5d2e4c1640 10 monclient(hunting): authenticate will time out at 7437.608428s 2024-03-25T18:23:58.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.399+0000 7f5d2ca5e640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5d2814be90 0x7f5d28142a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35218/0 (socket says 172.21.15.22:35218) 2024-03-25T18:23:58.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.399+0000 7f5d2ca5e640 1 -- 172.21.15.22:0/3893130316 learned_addr learned my addr 172.21.15.22:0/3893130316 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:23:58.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.401+0000 7f5d24ff9640 10 client.?.objecter ms_handle_connect 0x7f5d2814be90 2024-03-25T18:23:58.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.401+0000 7f5d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.408+0000 7f5d24ff9640 10 client.?.objecter ms_handle_connect 0x7f5d280b4d60 2024-03-25T18:23:58.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.408+0000 7f5d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.409+0000 7f5d24ff9640 10 client.?.objecter ms_handle_connect 0x7f5d280b1730 2024-03-25T18:23:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.409+0000 7f5d24ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.417+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363448085 0 0) 0x7f5d28141fa0 con 0x7f5d2814be90 2024-03-25T18:23:58.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.417+0000 7f5d24ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.417+0000 7f5d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.417+0000 7f5d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.417+0000 7f5d24ff9640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:23:58.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.417+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d14004eb0 con 0x7f5d2814be90 2024-03-25T18:23:58.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.424+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871556252 0 0) 0x7f5d28143130 con 0x7f5d280b1730 2024-03-25T18:23:58.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.424+0000 7f5d24ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.424+0000 7f5d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.424+0000 7f5d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.424+0000 7f5d24ff9640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:23:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.424+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d28141fa0 con 0x7f5d280b1730 2024-03-25T18:23:58.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.428+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344199409 0 0) 0x7f5d28140f70 con 0x7f5d280b4d60 2024-03-25T18:23:58.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.428+0000 7f5d24ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.428+0000 7f5d24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.428+0000 7f5d24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.428+0000 7f5d24ff9640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:23:58.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.428+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d28143130 con 0x7f5d280b4d60 2024-03-25T18:23:58.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.431+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3283828231 0 0) 0x7f5d14004eb0 con 0x7f5d2814be90 2024-03-25T18:23:58.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.431+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d28140f70 con 0x7f5d2814be90 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.431+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5d20001ec0 con 0x7f5d2814be90 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.431+0000 7f5d24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.431+0000 7f5d24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.431+0000 7f5d24ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.431+0000 7f5d24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.438+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2106075809 0 0) 0x7f5d28141fa0 con 0x7f5d280b1730 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.439+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d14004eb0 con 0x7f5d280b1730 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.439+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5d180018b0 con 0x7f5d280b1730 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.439+0000 7f5d24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.439+0000 7f5d24ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.439+0000 7f5d24ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.439 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:58.440 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:58.440 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:58.440 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:58.440 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.442+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649145066 0 0) 0x7f5d28143130 con 0x7f5d280b4d60 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.442+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d28141fa0 con 0x7f5d280b4d60 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.442+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5d1c001830 con 0x7f5d280b4d60 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.442+0000 7f5d24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.442+0000 7f5d24ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.442+0000 7f5d24ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.443 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:58.444 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:58.444 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:58.444 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:58.444 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063423259 0 0) 0x7f5d14004eb0 con 0x7f5d280b1730 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 >> v1:172.21.15.22:6790/0 conn(0x7f5d2814be90 legacy=0x7f5d28142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 >> v1:172.21.15.22:6789/0 conn(0x7f5d280b4d60 legacy=0x7f5d28142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 1 monclient: found mon.b 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d28137a90 con 0x7f5d280b1730 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 10 monclient: _renew_subs 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d281324b0 con 0x7f5d280b1730 2024-03-25T18:23:58.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 10 monclient: _renew_subs 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d28144ea0 con 0x7f5d280b1730 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.452+0000 7f5d2e4c1640 1 librados: init done 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.453+0000 7f5d2e4c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.453+0000 7f5d2e4c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.453+0000 7f5d2e4c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:58.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.453+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d280627b0 con 0x7f5d280b1730 2024-03-25T18:23:58.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d180035f0 con 0x7f5d280b1730 2024-03-25T18:23:58.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:58.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5d18003900 con 0x7f5d280b1730 2024-03-25T18:23:58.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d24ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d24ff9640 10 monclient: dump: 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:23:58.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d267fc640 4 set_mon_vals no callback set 2024-03-25T18:23:58.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.744+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f5d180198d0 con 0x7f5d280b1730 2024-03-25T18:23:58.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.745+0000 7f5d277fe640 10 monclient: get_auth_request con 0x7f5d1406e110 auth_method 0 2024-03-25T18:23:58.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.745+0000 7f5d24ff9640 10 client.4435.objecter ms_handle_connect 0x7f5d1406e110 2024-03-25T18:23:58.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.750+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f5d180535a0 con 0x7f5d280b1730 2024-03-25T18:23:58.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.750+0000 7f5d24ff9640 10 client.4435.objecter ms_dispatch 0x7f5d28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:58.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.750+0000 7f5d24ff9640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:58.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.750+0000 7f5d24ff9640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:58.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.751+0000 7f5d24ff9640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:23:58.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.752+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d1804eee0 con 0x7f5d280b1730 2024-03-25T18:23:58.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.752+0000 7f5d24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:58.752+0000 7f5d24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:59.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.003+0000 7f5d2e4c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:59.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.003+0000 7f5d2e4c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:59.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.003+0000 7f5d2e4c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:23:59.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.003+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d28137c20 con 0x7f5d280b1730 2024-03-25T18:23:59.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.026+0000 7f5d24ff9640 1 -- 172.21.15.22:0/3893130316 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f5d18050fe0 con 0x7f5d280b1730 2024-03-25T18:23:59.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.026+0000 7f5d24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:59.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.026+0000 7f5d24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:59.026 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:23:59.027 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 10 librados: watch_flush enter 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 10 librados: watch_flush exit 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 >> v1:172.21.15.33:6800/215350958 conn(0x7f5d1406e110 legacy=0x7f5d140705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 10 monclient: shutdown 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 >> v1:172.21.15.33:6789/0 conn(0x7f5d280b1730 legacy=0x7f5d28131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 shutdown_connections 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 >> 172.21.15.22:0/3893130316 conn(0x7f5d280617a0 msgr2=0x7f5d280b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:59.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 shutdown_connections 2024-03-25T18:23:59.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 1 -- 172.21.15.22:0/3893130316 wait complete. 2024-03-25T18:23:59.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:23:59.028+0000 7f5d2e4c1640 1 librados: shutdown 2024-03-25T18:23:59.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:05.846 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.959+0000 7ff8a6104640 10 monclient: get_monmap_and_config 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.959+0000 7ff8a6104640 10 monclient: build_initial_monmap 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.959+0000 7ff8a6104640 10 monclient: monmap: 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:05.960201+0000 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:05.960201+0000 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:05.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:05.961 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:05.961 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:05.961 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:05.961 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:05.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.960+0000 7ff8a6104640 1 Processor -- start 2024-03-25T18:24:05.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.960+0000 7ff8a6104640 1 -- start start 2024-03-25T18:24:05.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.960+0000 7ff8a6104640 10 monclient: init 2024-03-25T18:24:05.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:05.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a00adaf0 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a013fd20 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a0142a90 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a006fd30 con 0x7ff8a00adaf0 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a006e920 con 0x7ff8a013fd20 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00b01e0 con 0x7ff8a0142a90 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:05.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.961+0000 7ff8a6104640 10 monclient(hunting): authenticate will time out at 7445.184485s 2024-03-25T18:24:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.967+0000 7ff89effd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7ff8a013fd20 0x7ff8a0142380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:52754/0 (socket says 172.21.15.22:52754) 2024-03-25T18:24:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.967+0000 7ff89effd640 1 -- 172.21.15.22:0/2774369301 learned_addr learned my addr 172.21.15.22:0/2774369301 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:05.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.984+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654238038 0 0) 0x7ff8a006e920 con 0x7ff8a013fd20 2024-03-25T18:24:05.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.984+0000 7ff89e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:05.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.984+0000 7ff89e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:05.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.984+0000 7ff89e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:05.984 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.984+0000 7ff89e7fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:24:05.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.984+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff888003520 con 0x7ff8a013fd20 2024-03-25T18:24:05.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.991+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003836141 0 0) 0x7ff8a00b01e0 con 0x7ff8a0142a90 2024-03-25T18:24:05.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.991+0000 7ff89e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:05.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.991+0000 7ff89e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:05.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.991+0000 7ff89e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:05.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.991+0000 7ff89e7fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:24:05.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:05.991+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a006e920 con 0x7ff8a0142a90 2024-03-25T18:24:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.002+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2024467054 0 0) 0x7ff8a006fd30 con 0x7ff8a00adaf0 2024-03-25T18:24:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.002+0000 7ff89e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.002+0000 7ff89e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.002+0000 7ff89e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.002+0000 7ff89e7fc640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:24:06.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.002+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a00b01e0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2824258379 0 0) 0x7ff888003520 con 0x7ff8a013fd20 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 >> v1:172.21.15.22:6790/0 conn(0x7ff8a0142a90 legacy=0x7ff8a014dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 >> v1:172.21.15.22:6789/0 conn(0x7ff8a00adaf0 legacy=0x7ff8a013f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:24:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00b0630 con 0x7ff8a013fd20 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff894001690 con 0x7ff8a013fd20 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff8a6104640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff8a6104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff89e7fc640 10 monclient: dump: 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.006 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.007 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.007 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:06.007 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:06.007 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:06.007 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:06.007 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:06.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.004+0000 7ff8a6104640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:06.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8940018b0 con 0x7ff8a013fd20 2024-03-25T18:24:06.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff89e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:06.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff89e7fc640 1 -- 172.21.15.22:0/2774369301 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff894001bc0 con 0x7ff8a013fd20 2024-03-25T18:24:06.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff8a6104640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff8a6104640 4 set_mon_vals no callback set 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff89e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff89e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.028+0000 7ff89e7fc640 10 monclient: dump: 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:06.029 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.029+0000 7ff8a6104640 10 monclient: shutdown 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.029+0000 7ff8a6104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.029+0000 7ff8a6104640 1 -- 172.21.15.22:0/2774369301 >> v1:172.21.15.33:6789/0 conn(0x7ff8a013fd20 legacy=0x7ff8a0142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.029+0000 7ff8a6104640 1 -- 172.21.15.22:0/2774369301 shutdown_connections 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.029+0000 7ff8a6104640 1 -- 172.21.15.22:0/2774369301 >> 172.21.15.22:0/2774369301 conn(0x7ff8a00b0ad0 msgr2=0x7ff8a00b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.029+0000 7ff8a6104640 1 -- 172.21.15.22:0/2774369301 shutdown_connections 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.029+0000 7ff8a6104640 1 -- 172.21.15.22:0/2774369301 wait complete. 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.030+0000 7ff8a6104640 10 monclient: build_initial_monmap 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.030+0000 7ff8a6104640 10 monclient: monmap: 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:06.031 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.030+0000 7ff8a6104640 1 librados: starting msgr at 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.030+0000 7ff8a6104640 1 librados: starting objecter 2024-03-25T18:24:06.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.031+0000 7ff8a6104640 1 Processor -- start 2024-03-25T18:24:06.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.031+0000 7ff8a6104640 1 -- start start 2024-03-25T18:24:06.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.031+0000 7ff8a6104640 1 librados: setting wanted keys 2024-03-25T18:24:06.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.031+0000 7ff8a6104640 1 librados: calling monclient init 2024-03-25T18:24:06.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.031+0000 7ff8a6104640 10 monclient: init 2024-03-25T18:24:06.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 10 monclient(hunting): picked mon.noname-c con 0x7ff8a00adaf0 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 10 monclient(hunting): picked mon.noname-b con 0x7ff8a013fd20 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 10 monclient(hunting): picked mon.noname-a con 0x7ff8a0142a90 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a01cb460 con 0x7ff8a0142a90 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a01cb740 con 0x7ff8a013fd20 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a01c9350 con 0x7ff8a00adaf0 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.032+0000 7ff8a6104640 10 monclient(hunting): authenticate will time out at 7445.255011s 2024-03-25T18:24:06.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.041+0000 7ff89f7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff8a00adaf0 0x7ff8a01c9c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:33122/0 (socket says 172.21.15.22:33122) 2024-03-25T18:24:06.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.041+0000 7ff89f7fe640 1 -- 172.21.15.22:0/126494257 learned_addr learned my addr 172.21.15.22:0/126494257 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:06.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.046+0000 7ff887fff640 10 client.?.objecter ms_handle_connect 0x7ff8a00adaf0 2024-03-25T18:24:06.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.046+0000 7ff887fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:06.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135975479 0 0) 0x7ff8a01c9350 con 0x7ff8a00adaf0 2024-03-25T18:24:06.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:06.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:06.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:06.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:24:06.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff88c004eb0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 10 client.?.objecter ms_handle_connect 0x7ff8a013fd20 2024-03-25T18:24:06.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.055+0000 7ff887fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:06.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.064+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908173127 0 0) 0x7ff88c004eb0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.064+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8a01c9350 con 0x7ff8a00adaf0 2024-03-25T18:24:06.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.064+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff890001a10 con 0x7ff8a00adaf0 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.064+0000 7ff887fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.064+0000 7ff887fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.064+0000 7ff887fff640 10 monclient(hunting): dump: 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:06.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.064+0000 7ff887fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:06.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.071+0000 7ff887fff640 10 client.?.objecter ms_handle_connect 0x7ff8a0142a90 2024-03-25T18:24:06.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.071+0000 7ff887fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:06.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1730419280 0 0) 0x7ff8a01c9350 con 0x7ff8a00adaf0 2024-03-25T18:24:06.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 >> v1:172.21.15.33:6789/0 conn(0x7ff8a013fd20 legacy=0x7ff8a01d7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 >> v1:172.21.15.22:6789/0 conn(0x7ff8a0142a90 legacy=0x7ff8a01d80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:06.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 1 monclient: found mon.c 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a01c94e0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff8a6104640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff8a6104640 10 monclient: _renew_subs 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff8a6104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a01e32e0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.072+0000 7ff8a6104640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.073+0000 7ff8a6104640 10 monclient: _renew_subs 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.073+0000 7ff8a6104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:06.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.073+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a01e3650 con 0x7ff8a00adaf0 2024-03-25T18:24:06.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.073+0000 7ff8a6104640 1 librados: init done 2024-03-25T18:24:06.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.074+0000 7ff8a6104640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:06.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.074+0000 7ff8a6104640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:06.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.074+0000 7ff8a6104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:06.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.074+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a0074d20 con 0x7ff8a00adaf0 2024-03-25T18:24:06.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.385+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff890001cf0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.385+0000 7ff887fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:06.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.385+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff890003a60 con 0x7ff8a00adaf0 2024-03-25T18:24:06.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.385+0000 7ff887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.385+0000 7ff887fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.385+0000 7ff887fff640 10 monclient: dump: 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:06.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.385+0000 7ff89dffb640 4 set_mon_vals no callback set 2024-03-25T18:24:06.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.386+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7ff890003c80 con 0x7ff8a00adaf0 2024-03-25T18:24:06.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.386+0000 7ff89effd640 10 monclient: get_auth_request con 0x7ff88c06dd00 auth_method 0 2024-03-25T18:24:06.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.387+0000 7ff887fff640 10 client.4442.objecter ms_handle_connect 0x7ff88c06dd00 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7ff8900548c0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 10 client.4442.objecter ms_dispatch 0x7ff8a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8900503d0 con 0x7ff8a00adaf0 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:06.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.393+0000 7ff887fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:06.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.645+0000 7ff8a6104640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.645+0000 7ff8a6104640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.645+0000 7ff8a6104640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:06.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.645+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8a006e000 con 0x7ff8a00adaf0 2024-03-25T18:24:06.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.676+0000 7ff887fff640 1 -- 172.21.15.22:0/126494257 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7ff8a006e000 con 0x7ff8a00adaf0 2024-03-25T18:24:06.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.676+0000 7ff887fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.676+0000 7ff887fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:06.677 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:24:06.677 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:06.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 10 librados: watch_flush enter 2024-03-25T18:24:06.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 10 librados: watch_flush exit 2024-03-25T18:24:06.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 10 client.4442.objecter successfully canceled tick 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 >> v1:172.21.15.33:6800/215350958 conn(0x7ff88c06dd00 legacy=0x7ff88c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 10 monclient: shutdown 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 >> v1:172.21.15.22:6790/0 conn(0x7ff8a00adaf0 legacy=0x7ff8a01c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 shutdown_connections 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.678+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 >> 172.21.15.22:0/126494257 conn(0x7ff8a00b0ad0 msgr2=0x7ff8a0139cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.679+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 shutdown_connections 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.679+0000 7ff8a6104640 1 -- 172.21.15.22:0/126494257 wait complete. 2024-03-25T18:24:06.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:06.679+0000 7ff8a6104640 1 librados: shutdown 2024-03-25T18:24:06.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:13.163 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:05:15.825 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.178 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:05:15.839 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.209 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:05:15.871 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.222 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:05:15.884 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.227 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:05:15.845 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.233 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:05:15.851 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.235 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:05:15.897 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.262 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:05:15.880 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.269 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:05:15.931 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:13.496 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.612+0000 7f184dc3e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.612+0000 7f184dc3e640 10 monclient: build_initial_monmap 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.612+0000 7f184dc3e640 10 monclient: monmap: 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:13.614177+0000 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:13.614177+0000 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:13.614 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:13.615 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.613+0000 7f184dc3e640 1 Processor -- start 2024-03-25T18:24:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.613+0000 7f184dc3e640 1 -- start start 2024-03-25T18:24:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.613+0000 7f184dc3e640 10 monclient: init 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f18480b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f18480b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f184814be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480af940 con 0x7f18480b1730 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480b0a70 con 0x7f184814be90 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18480b0d50 con 0x7f18480b4d60 2024-03-25T18:24:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:13.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.614+0000 7f184dc3e640 10 monclient(hunting): authenticate will time out at 7452.838503s 2024-03-25T18:24:13.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.618+0000 7f1846ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f18480b1730 0x7f18480b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:59752/0 (socket says 172.21.15.22:59752) 2024-03-25T18:24:13.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.618+0000 7f1846ffd640 1 -- 172.21.15.22:0/3618935575 learned_addr learned my addr 172.21.15.22:0/3618935575 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:13.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.635+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490543397 0 0) 0x7f18480af940 con 0x7f18480b1730 2024-03-25T18:24:13.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.635+0000 7f18467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.635+0000 7f18467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.635+0000 7f18467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.635+0000 7f18467fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:24:13.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.635+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1830003520 con 0x7f18480b1730 2024-03-25T18:24:13.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843471157 0 0) 0x7f18480b0a70 con 0x7f184814be90 2024-03-25T18:24:13.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:24:13.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18480af940 con 0x7f184814be90 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3950318706 0 0) 0x7f1830003520 con 0x7f18480b1730 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 >> v1:172.21.15.22:6790/0 conn(0x7f18480b4d60 legacy=0x7f184813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 >> v1:172.21.15.33:6789/0 conn(0x7f184814be90 legacy=0x7f184814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 monclient: found mon.noname-a 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18480b1030 con 0x7f18480b1730 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f183c001690 con 0x7f18480b1730 2024-03-25T18:24:13.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.648+0000 7f18467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.649+0000 7f18467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.649+0000 7f18467fc640 10 monclient: dump: 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.649+0000 7f184dc3e640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:24:13.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.649+0000 7f184dc3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f183c0017f0 con 0x7f18480b1730 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f18467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f18467fc640 1 -- 172.21.15.22:0/3618935575 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f183c001c00 con 0x7f18480b1730 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f184dc3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f184dc3e640 4 set_mon_vals no callback set 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f18467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f18467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:13.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f18467fc640 10 monclient: dump: 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:13.662 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f184dc3e640 10 monclient: shutdown 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f184dc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.660+0000 7f184dc3e640 1 -- 172.21.15.22:0/3618935575 >> v1:172.21.15.22:6789/0 conn(0x7f18480b1730 legacy=0x7f18480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.661+0000 7f184dc3e640 1 -- 172.21.15.22:0/3618935575 shutdown_connections 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.661+0000 7f184dc3e640 1 -- 172.21.15.22:0/3618935575 >> 172.21.15.22:0/3618935575 conn(0x7f18480617a0 msgr2=0x7f1848061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.661+0000 7f184dc3e640 1 -- 172.21.15.22:0/3618935575 shutdown_connections 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.661+0000 7f184dc3e640 1 -- 172.21.15.22:0/3618935575 wait complete. 2024-03-25T18:24:13.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 10 monclient: build_initial_monmap 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 10 monclient: monmap: 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:13.664 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 1 librados: starting msgr at 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 1 librados: starting objecter 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 1 Processor -- start 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 1 -- start start 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 1 librados: setting wanted keys 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 1 librados: calling monclient init 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.662+0000 7f184dc3e640 10 monclient: init 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 10 monclient(hunting): picked mon.noname-a con 0x7f18480b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:13.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 10 monclient(hunting): picked mon.noname-b con 0x7f18480b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:13.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 10 monclient(hunting): picked mon.noname-c con 0x7f184814be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:13.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1848137fa0 con 0x7f18480b1730 2024-03-25T18:24:13.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18481361c0 con 0x7f18480b4d60 2024-03-25T18:24:13.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1848135ca0 con 0x7f184814be90 2024-03-25T18:24:13.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:13.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.663+0000 7f184dc3e640 10 monclient(hunting): authenticate will time out at 7452.887463s 2024-03-25T18:24:13.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.668+0000 7f1847fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f184814be90 0x7f18481e2f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58322/0 (socket says 172.21.15.22:58322) 2024-03-25T18:24:13.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.668+0000 7f1847fff640 1 -- 172.21.15.22:0/1802349291 learned_addr learned my addr 172.21.15.22:0/1802349291 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:13.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.668+0000 7f182ffff640 10 client.?.objecter ms_handle_connect 0x7f184814be90 2024-03-25T18:24:13.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.669+0000 7f182ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.675+0000 7f182ffff640 10 client.?.objecter ms_handle_connect 0x7f18480b4d60 2024-03-25T18:24:13.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.675+0000 7f182ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.680+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037723847 0 0) 0x7f1848135ca0 con 0x7f184814be90 2024-03-25T18:24:13.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.680+0000 7f182ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.680+0000 7f182ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.680+0000 7f182ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.680+0000 7f182ffff640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:24:13.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.680+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1834004eb0 con 0x7f184814be90 2024-03-25T18:24:13.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.682+0000 7f182ffff640 10 client.?.objecter ms_handle_connect 0x7f18480b1730 2024-03-25T18:24:13.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.682+0000 7f182ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.691+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232113253 0 0) 0x7f1848137fa0 con 0x7f18480b1730 2024-03-25T18:24:13.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.691+0000 7f182ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.691+0000 7f182ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.691+0000 7f182ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.691+0000 7f182ffff640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:24:13.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.691+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848135ca0 con 0x7f18480b1730 2024-03-25T18:24:13.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.693+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2718267021 0 0) 0x7f1834004eb0 con 0x7f184814be90 2024-03-25T18:24:13.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.693+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1848137fa0 con 0x7f184814be90 2024-03-25T18:24:13.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.693+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f1840001c90 con 0x7f184814be90 2024-03-25T18:24:13.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.693+0000 7f182ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.693+0000 7f182ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.693+0000 7f182ffff640 10 monclient(hunting): dump: 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:13.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.693+0000 7f182ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.700+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3575845473 0 0) 0x7f1848135ca0 con 0x7f18480b1730 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.701+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1834004eb0 con 0x7f18480b1730 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.701+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f18380018b0 con 0x7f18480b1730 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.701+0000 7f182ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.701+0000 7f182ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.701+0000 7f182ffff640 10 monclient(hunting): dump: 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:13.702 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:13.703 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4171360014 0 0) 0x7f1848137fa0 con 0x7f184814be90 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 >> v1:172.21.15.33:6789/0 conn(0x7f18480b4d60 legacy=0x7f18481dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 >> v1:172.21.15.22:6789/0 conn(0x7f18480b1730 legacy=0x7f184812fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 1 monclient: found mon.c 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1848138350 con 0x7f184814be90 2024-03-25T18:24:13.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f182ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 5 monclient: authenticate success, global_id 4448 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 10 monclient: _renew_subs 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1848136410 con 0x7f184814be90 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 10 monclient: _renew_subs 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1848130770 con 0x7f184814be90 2024-03-25T18:24:13.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.705+0000 7f184dc3e640 1 librados: init done 2024-03-25T18:24:13.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.706+0000 7f184dc3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:13.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.706+0000 7f184dc3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:13.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.706+0000 7f184dc3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:13.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:13.706+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1848135e30 con 0x7f184814be90 2024-03-25T18:24:14.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.028+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1840003990 con 0x7f184814be90 2024-03-25T18:24:14.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.028+0000 7f182ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:14.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.028+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f1840003d20 con 0x7f184814be90 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.028+0000 7f182ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.028+0000 7f182ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.028+0000 7f182ffff640 10 monclient: dump: 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:14.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.028+0000 7f1845ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:14.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.029+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f1840003f40 con 0x7f184814be90 2024-03-25T18:24:14.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.029+0000 7f18477fe640 10 monclient: get_auth_request con 0x7f183406de90 auth_method 0 2024-03-25T18:24:14.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.029+0000 7f182ffff640 10 client.4448.objecter ms_handle_connect 0x7f183406de90 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f1840054a10 con 0x7f184814be90 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 10 client.4448.objecter ms_dispatch 0x7f1848000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1840050520 con 0x7f184814be90 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:14.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.037+0000 7f182ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:14.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.273+0000 7f184dc3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.273+0000 7f184dc3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.273+0000 7f184dc3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:14.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.273+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18481365a0 con 0x7f184814be90 2024-03-25T18:24:14.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.294+0000 7f182ffff640 1 -- 172.21.15.22:0/1802349291 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f1840052450 con 0x7f184814be90 2024-03-25T18:24:14.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.294+0000 7f182ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:14.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.294+0000 7f182ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:14.296 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:24:14.296 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:14.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.296+0000 7f184dc3e640 10 librados: watch_flush enter 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.296+0000 7f184dc3e640 10 librados: watch_flush exit 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.296+0000 7f184dc3e640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.296+0000 7f184dc3e640 10 client.4448.objecter successfully canceled tick 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.296+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 >> v1:172.21.15.33:6800/215350958 conn(0x7f183406de90 legacy=0x7f1834070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 10 monclient: shutdown 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 >> v1:172.21.15.22:6790/0 conn(0x7f184814be90 legacy=0x7f18481e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 shutdown_connections 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 >> 172.21.15.22:0/1802349291 conn(0x7f18480617a0 msgr2=0x7f18480b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 shutdown_connections 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 1 -- 172.21.15.22:0/1802349291 wait complete. 2024-03-25T18:24:14.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:14.297+0000 7f184dc3e640 1 librados: shutdown 2024-03-25T18:24:14.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:21.117 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:21.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.235+0000 7efdd535d640 10 monclient: get_monmap_and_config 2024-03-25T18:24:21.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.235+0000 7efdd535d640 10 monclient: build_initial_monmap 2024-03-25T18:24:21.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.235+0000 7efdd535d640 10 monclient: monmap: 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:21.237161+0000 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:21.237161+0000 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.236+0000 7efdd535d640 1 Processor -- start 2024-03-25T18:24:21.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.236+0000 7efdd535d640 1 -- start start 2024-03-25T18:24:21.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.236+0000 7efdd535d640 10 monclient: init 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 10 monclient(hunting): picked mon.noname-a con 0x7efdd00a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 10 monclient(hunting): picked mon.noname-b con 0x7efdd013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 10 monclient(hunting): picked mon.noname-c con 0x7efdd01430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd0142520 con 0x7efdd00a98f0 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00b28a0 con 0x7efdd01430c0 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd00b2a30 con 0x7efdd013f710 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.237+0000 7efdd535d640 10 monclient(hunting): authenticate will time out at 7460.461116s 2024-03-25T18:24:21.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.240+0000 7efdceffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7efdd00a98f0 0x7efdd00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36074/0 (socket says 172.21.15.22:36074) 2024-03-25T18:24:21.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.240+0000 7efdceffd640 1 -- 172.21.15.22:0/918731319 learned_addr learned my addr 172.21.15.22:0/918731319 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:21.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.253+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3516833225 0 0) 0x7efdd00b28a0 con 0x7efdd01430c0 2024-03-25T18:24:21.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.253+0000 7efdcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.253+0000 7efdcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.253+0000 7efdcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.253+0000 7efdcdffb640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:24:21.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.253+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdb8003520 con 0x7efdd01430c0 2024-03-25T18:24:21.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.257+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127600586 0 0) 0x7efdd0142520 con 0x7efdd00a98f0 2024-03-25T18:24:21.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.257+0000 7efdcdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.257+0000 7efdcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.257+0000 7efdcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.257+0000 7efdcdffb640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:24:21.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.258+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd00b28a0 con 0x7efdd00a98f0 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2666732985 0 0) 0x7efdd00b28a0 con 0x7efdd00a98f0 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 >> v1:172.21.15.22:6790/0 conn(0x7efdd013f710 legacy=0x7efdd01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 >> v1:172.21.15.33:6789/0 conn(0x7efdd01430c0 legacy=0x7efdd014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 1 monclient: found mon.noname-a 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:24:21.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd00b44c0 con 0x7efdd00a98f0 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efdc0001670 con 0x7efdd00a98f0 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdd535d640 5 monclient: authenticate success, global_id 4446 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdd535d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdcdffb640 10 monclient: dump: 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:21.265 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.262+0000 7efdd535d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.265+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc00017d0 con 0x7efdd00a98f0 2024-03-25T18:24:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.265+0000 7efdcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdcdffb640 1 -- 172.21.15.22:0/918731319 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efdc0001ba0 con 0x7efdd00a98f0 2024-03-25T18:24:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 4 set_mon_vals no callback set 2024-03-25T18:24:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdcdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdcdffb640 10 monclient: dump: 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:21.268 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 10 monclient: shutdown 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 1 -- 172.21.15.22:0/918731319 >> v1:172.21.15.22:6789/0 conn(0x7efdd00a98f0 legacy=0x7efdd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 1 -- 172.21.15.22:0/918731319 shutdown_connections 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 1 -- 172.21.15.22:0/918731319 >> 172.21.15.22:0/918731319 conn(0x7efdd00b7ea0 msgr2=0x7efdd00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 1 -- 172.21.15.22:0/918731319 shutdown_connections 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.266+0000 7efdd535d640 1 -- 172.21.15.22:0/918731319 wait complete. 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.267+0000 7efdd535d640 10 monclient: build_initial_monmap 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.267+0000 7efdd535d640 10 monclient: monmap: 2024-03-25T18:24:21.271 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 1 librados: starting msgr at 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 1 librados: starting objecter 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 1 Processor -- start 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 1 -- start start 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 1 librados: setting wanted keys 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 1 librados: calling monclient init 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 10 monclient: init 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.268+0000 7efdd535d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 10 monclient(hunting): picked mon.noname-a con 0x7efdd00a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 10 monclient(hunting): picked mon.noname-c con 0x7efdd013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 10 monclient(hunting): picked mon.noname-b con 0x7efdd01430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd01da8f0 con 0x7efdd00a98f0 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd01399f0 con 0x7efdd01430c0 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdd01db770 con 0x7efdd013f710 2024-03-25T18:24:21.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:21.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.269+0000 7efdd535d640 10 monclient(hunting): authenticate will time out at 7460.492733s 2024-03-25T18:24:21.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.272+0000 7efdcf7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7efdd01430c0 0x7efdd01e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:59134/0 (socket says 172.21.15.22:59134) 2024-03-25T18:24:21.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.272+0000 7efdcf7fe640 1 -- 172.21.15.22:0/3179980179 learned_addr learned my addr 172.21.15.22:0/3179980179 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:21.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.273+0000 7efdb77fe640 10 client.?.objecter ms_handle_connect 0x7efdd01430c0 2024-03-25T18:24:21.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.273+0000 7efdb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:21.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.280+0000 7efdb77fe640 10 client.?.objecter ms_handle_connect 0x7efdd00a98f0 2024-03-25T18:24:21.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.280+0000 7efdb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:21.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.282+0000 7efdb77fe640 10 client.?.objecter ms_handle_connect 0x7efdd013f710 2024-03-25T18:24:21.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.282+0000 7efdb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.289+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061913173 0 0) 0x7efdd01399f0 con 0x7efdd01430c0 2024-03-25T18:24:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.289+0000 7efdb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.289+0000 7efdb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.289+0000 7efdb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.289+0000 7efdb77fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:24:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.289+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdbc004eb0 con 0x7efdd01430c0 2024-03-25T18:24:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747230775 0 0) 0x7efdd01db770 con 0x7efdd013f710 2024-03-25T18:24:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:24:21.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd01399f0 con 0x7efdd013f710 2024-03-25T18:24:21.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965014397 0 0) 0x7efdd01da8f0 con 0x7efdd00a98f0 2024-03-25T18:24:21.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:24:21.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.293+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd01db770 con 0x7efdd00a98f0 2024-03-25T18:24:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.300+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3308256990 0 0) 0x7efdbc004eb0 con 0x7efdd01430c0 2024-03-25T18:24:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.300+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdd01da8f0 con 0x7efdd01430c0 2024-03-25T18:24:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.300+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efdc8001c70 con 0x7efdd01430c0 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.300+0000 7efdb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.300+0000 7efdb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.300+0000 7efdb77fe640 10 monclient(hunting): dump: 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:21.302 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.300+0000 7efdb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:21.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.313+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3620954612 0 0) 0x7efdd01399f0 con 0x7efdd013f710 2024-03-25T18:24:21.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.313+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdbc004eb0 con 0x7efdd013f710 2024-03-25T18:24:21.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.313+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efdc40018a0 con 0x7efdd013f710 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.313+0000 7efdb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.313+0000 7efdb77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.313+0000 7efdb77fe640 10 monclient(hunting): dump: 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:21.316 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.314+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1285303326 0 0) 0x7efdd01db770 con 0x7efdd00a98f0 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.314+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efdd01399f0 con 0x7efdd00a98f0 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.314+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efdc0001ae0 con 0x7efdd00a98f0 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.314+0000 7efdb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.314+0000 7efdb77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.314+0000 7efdb77fe640 10 monclient(hunting): dump: 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.317 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:21.318 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:21.318 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:21.318 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:21.318 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1135139183 0 0) 0x7efdd01da8f0 con 0x7efdd01430c0 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 >> v1:172.21.15.22:6790/0 conn(0x7efdd013f710 legacy=0x7efdd01381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 >> v1:172.21.15.22:6789/0 conn(0x7efdd00a98f0 legacy=0x7efdd01db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 1 monclient: found mon.b 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd01dab40 con 0x7efdd01430c0 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:21.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 5 monclient: authenticate success, global_id 4453 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 10 monclient: _renew_subs 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd0137870 con 0x7efdd01430c0 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 10 monclient: _renew_subs 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd0138ae0 con 0x7efdd01430c0 2024-03-25T18:24:21.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.325+0000 7efdd535d640 1 librados: init done 2024-03-25T18:24:21.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.326+0000 7efdd535d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:21.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.326+0000 7efdd535d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:21.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.326+0000 7efdd535d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:21.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.326+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdd00627b0 con 0x7efdd01430c0 2024-03-25T18:24:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.617+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efdc8003990 con 0x7efdd01430c0 2024-03-25T18:24:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.617+0000 7efdb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.617+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7efdc8003da0 con 0x7efdd01430c0 2024-03-25T18:24:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.617+0000 7efdb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.617+0000 7efdb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.617+0000 7efdb77fe640 10 monclient: dump: 2024-03-25T18:24:21.619 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.617+0000 7efdcd7fa640 4 set_mon_vals no callback set 2024-03-25T18:24:21.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.618+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7efdc8003fc0 con 0x7efdd01430c0 2024-03-25T18:24:21.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.618+0000 7efdceffd640 10 monclient: get_auth_request con 0x7efdbc06e0f0 auth_method 0 2024-03-25T18:24:21.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.619+0000 7efdb77fe640 10 client.4453.objecter ms_handle_connect 0x7efdbc06e0f0 2024-03-25T18:24:21.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7efdc8054a90 con 0x7efdd01430c0 2024-03-25T18:24:21.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 10 client.4453.objecter ms_dispatch 0x7efdd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:21.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:21.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:21.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T18:24:21.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efdc80505a0 con 0x7efdd01430c0 2024-03-25T18:24:21.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:21.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.624+0000 7efdb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:21.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.877+0000 7efdd535d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:21.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.877+0000 7efdd535d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:21.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.877+0000 7efdd535d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:21.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.877+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efdd0069f40 con 0x7efdd01430c0 2024-03-25T18:24:21.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.941+0000 7efdb77fe640 1 -- 172.21.15.22:0/3179980179 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7efdc80524d0 con 0x7efdd01430c0 2024-03-25T18:24:21.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.941+0000 7efdb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:21.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.941+0000 7efdb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:21.943 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:24:21.943 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:21.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 10 librados: watch_flush enter 2024-03-25T18:24:21.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 10 librados: watch_flush exit 2024-03-25T18:24:21.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T18:24:21.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 10 client.4453.objecter successfully canceled tick 2024-03-25T18:24:21.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 >> v1:172.21.15.33:6800/215350958 conn(0x7efdbc06e0f0 legacy=0x7efdbc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 10 monclient: shutdown 2024-03-25T18:24:21.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:21.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 >> v1:172.21.15.33:6789/0 conn(0x7efdd01430c0 legacy=0x7efdd01e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 shutdown_connections 2024-03-25T18:24:21.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 >> 172.21.15.22:0/3179980179 conn(0x7efdd00b7ea0 msgr2=0x7efdd013d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:21.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 shutdown_connections 2024-03-25T18:24:21.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 1 -- 172.21.15.22:0/3179980179 wait complete. 2024-03-25T18:24:21.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:21.943+0000 7efdd535d640 1 librados: shutdown 2024-03-25T18:24:21.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:28.764 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.876+0000 7f82a336b640 10 monclient: get_monmap_and_config 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.876+0000 7f82a336b640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.876+0000 7f82a336b640 10 monclient: monmap: 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:28.877947+0000 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:28.877947+0000 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.878 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:28.879 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:28.879 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:28.879 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:28.879 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:28.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.877+0000 7f82a336b640 1 Processor -- start 2024-03-25T18:24:28.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.877+0000 7f82a336b640 1 -- start start 2024-03-25T18:24:28.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.877+0000 7f82a336b640 10 monclient: init 2024-03-25T18:24:28.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.878+0000 7f82a336b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.878+0000 7f82a336b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:28.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.878+0000 7f82a336b640 10 monclient(hunting): picked mon.noname-c con 0x7f829c0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:28.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.878+0000 7f82a336b640 10 monclient(hunting): picked mon.noname-a con 0x7f829c13f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:28.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.879+0000 7f82a336b640 10 monclient(hunting): picked mon.noname-b con 0x7f829c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:28.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.879+0000 7f82a336b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c142520 con 0x7f829c13f710 2024-03-25T18:24:28.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.879+0000 7f82a336b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0b28a0 con 0x7f829c0a98f0 2024-03-25T18:24:28.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.879+0000 7f82a336b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0b2a30 con 0x7f829c1430c0 2024-03-25T18:24:28.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.879+0000 7f82a336b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.879+0000 7f82a336b640 10 monclient(hunting): authenticate will time out at 7468.102739s 2024-03-25T18:24:28.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.883+0000 7f82a1107640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f829c0a98f0 0x7f829c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:43918/0 (socket says 172.21.15.22:43918) 2024-03-25T18:24:28.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.883+0000 7f82a1107640 1 -- 172.21.15.22:0/3245357805 learned_addr learned my addr 172.21.15.22:0/3245357805 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440272020 0 0) 0x7f829c0b28a0 con 0x7f829c0a98f0 2024-03-25T18:24:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:24:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8288003520 con 0x7f829c0a98f0 2024-03-25T18:24:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217659850 0 0) 0x7f829c142520 con 0x7f829c13f710 2024-03-25T18:24:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:24:28.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.896+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0b28a0 con 0x7f829c13f710 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951671916 0 0) 0x7f829c0b2a30 con 0x7f829c1430c0 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c142520 con 0x7f829c1430c0 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3725155782 0 0) 0x7f8288003520 con 0x7f829c0a98f0 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 >> v1:172.21.15.22:6790/0 conn(0x7f829c1430c0 legacy=0x7f829c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 >> v1:172.21.15.22:6789/0 conn(0x7f829c13f710 legacy=0x7f829c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 monclient: found mon.noname-c 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0b44c0 con 0x7f829c0a98f0 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8298001690 con 0x7f829c0a98f0 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f8293fff640 10 monclient: dump: 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f82a336b640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:24:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.909+0000 7f82a336b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:28.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82980017f0 con 0x7f829c0a98f0 2024-03-25T18:24:28.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f8293fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f8293fff640 1 -- 172.21.15.22:0/3245357805 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8298001c00 con 0x7f829c0a98f0 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f82a336b640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f82a336b640 4 set_mon_vals no callback set 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f8293fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f8293fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.919+0000 7f8293fff640 10 monclient: dump: 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:28.921 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:28.922 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:28.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.920+0000 7f82a336b640 10 monclient: shutdown 2024-03-25T18:24:28.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.920+0000 7f82a336b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.920+0000 7f82a336b640 1 -- 172.21.15.22:0/3245357805 >> v1:172.21.15.33:6789/0 conn(0x7f829c0a98f0 legacy=0x7f829c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.920+0000 7f82a336b640 1 -- 172.21.15.22:0/3245357805 shutdown_connections 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.920+0000 7f82a336b640 1 -- 172.21.15.22:0/3245357805 >> 172.21.15.22:0/3245357805 conn(0x7f829c0b7ea0 msgr2=0x7f829c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.920+0000 7f82a336b640 1 -- 172.21.15.22:0/3245357805 shutdown_connections 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.920+0000 7f82a336b640 1 -- 172.21.15.22:0/3245357805 wait complete. 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.921+0000 7f82a336b640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.921+0000 7f82a336b640 10 monclient: monmap: 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:28.923 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:28.924 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:28.924 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:28.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.921+0000 7f82a336b640 1 librados: starting msgr at 2024-03-25T18:24:28.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.921+0000 7f82a336b640 1 librados: starting objecter 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 1 Processor -- start 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 1 -- start start 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 1 librados: setting wanted keys 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 1 librados: calling monclient init 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 10 monclient: init 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 10 monclient(hunting): picked mon.noname-a con 0x7f829c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 10 monclient(hunting): picked mon.noname-b con 0x7f829c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.922+0000 7f82a336b640 10 monclient(hunting): picked mon.noname-c con 0x7f829c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.923+0000 7f82a336b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c06f330 con 0x7f829c0a98f0 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.923+0000 7f82a336b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0ac0a0 con 0x7f829c13f710 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.923+0000 7f82a336b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f829c0ac310 con 0x7f829c1430c0 2024-03-25T18:24:28.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.923+0000 7f82a336b640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.923+0000 7f82a336b640 10 monclient(hunting): authenticate will time out at 7468.146668s 2024-03-25T18:24:28.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.933+0000 7f82a0906640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f829c13f710 0x7f829c1e38c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:43954/0 (socket says 172.21.15.22:43954) 2024-03-25T18:24:28.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.933+0000 7f82a0906640 1 -- 172.21.15.22:0/1183501827 learned_addr learned my addr 172.21.15.22:0/1183501827 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:28.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.934+0000 7f8291ffb640 10 client.?.objecter ms_handle_connect 0x7f829c13f710 2024-03-25T18:24:28.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.934+0000 7f8291ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.938+0000 7f8291ffb640 10 client.?.objecter ms_handle_connect 0x7f829c0a98f0 2024-03-25T18:24:28.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.938+0000 7f8291ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.942+0000 7f8291ffb640 10 client.?.objecter ms_handle_connect 0x7f829c1430c0 2024-03-25T18:24:28.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.942+0000 7f8291ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.944+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192607097 0 0) 0x7f829c0ac0a0 con 0x7f829c13f710 2024-03-25T18:24:28.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.944+0000 7f8291ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.944+0000 7f8291ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.944+0000 7f8291ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.944+0000 7f8291ffb640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:24:28.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.944+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8280004eb0 con 0x7f829c13f710 2024-03-25T18:24:28.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.950+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069109560 0 0) 0x7f829c0ac310 con 0x7f829c1430c0 2024-03-25T18:24:28.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.950+0000 7f8291ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.950+0000 7f8291ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.950+0000 7f8291ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.950+0000 7f8291ffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:24:28.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.950+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0ac0a0 con 0x7f829c1430c0 2024-03-25T18:24:28.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.951+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717476646 0 0) 0x7f829c06f330 con 0x7f829c0a98f0 2024-03-25T18:24:28.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.951+0000 7f8291ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.951+0000 7f8291ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.951+0000 7f8291ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.951+0000 7f8291ffb640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:24:28.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.951+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0ac310 con 0x7f829c0a98f0 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.959+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1372633288 0 0) 0x7f8280004eb0 con 0x7f829c13f710 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.959+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f829c06f330 con 0x7f829c13f710 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.959+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f828c001a50 con 0x7f829c13f710 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.959+0000 7f8291ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.959+0000 7f8291ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.959+0000 7f8291ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:28.961 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:28.962 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:28.962 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:28.962 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:28.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.959+0000 7f8291ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:28.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.966+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290849886 0 0) 0x7f829c0ac0a0 con 0x7f829c1430c0 2024-03-25T18:24:28.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.966+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8280004eb0 con 0x7f829c1430c0 2024-03-25T18:24:28.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.966+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8294001b50 con 0x7f829c1430c0 2024-03-25T18:24:28.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.966+0000 7f8291ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.966+0000 7f8291ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.966+0000 7f8291ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:28.968 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.967+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1012417738 0 0) 0x7f829c0ac310 con 0x7f829c0a98f0 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.967+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f829c0ac0a0 con 0x7f829c0a98f0 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.967+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f8298001b00 con 0x7f829c0a98f0 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.967+0000 7f8291ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.967+0000 7f8291ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.967+0000 7f8291ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.969 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:28.970 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.970 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:28.970 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:28.970 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:28.970 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:28.970 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2024680077 0 0) 0x7f829c06f330 con 0x7f829c13f710 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 >> v1:172.21.15.22:6790/0 conn(0x7f829c1430c0 legacy=0x7f829c1e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 >> v1:172.21.15.22:6789/0 conn(0x7f829c0a98f0 legacy=0x7f829c071420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 1 monclient: found mon.b 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c0ac5f0 con 0x7f829c13f710 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f8291ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 10 monclient: _renew_subs 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c071b30 con 0x7f829c13f710 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 10 monclient: _renew_subs 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c06f9a0 con 0x7f829c13f710 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.970+0000 7f82a336b640 1 librados: init done 2024-03-25T18:24:28.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.971+0000 7f82a336b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.971+0000 7f82a336b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.971+0000 7f82a336b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:28.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:28.971+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c074cc0 con 0x7f829c13f710 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.258+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f828c001c70 con 0x7f829c13f710 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.258+0000 7f8291ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.258+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f828c003a60 con 0x7f829c13f710 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.258+0000 7f8291ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.258+0000 7f8291ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.258+0000 7f8291ffb640 10 monclient: dump: 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.260 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:29.261 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:29.261 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:29.261 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:29.261 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:29.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.258+0000 7f82937fe640 4 set_mon_vals no callback set 2024-03-25T18:24:29.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.259+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f828c003c40 con 0x7f829c13f710 2024-03-25T18:24:29.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.260+0000 7f82a1107640 10 monclient: get_auth_request con 0x7f828006e0f0 auth_method 0 2024-03-25T18:24:29.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.260+0000 7f8291ffb640 10 client.4459.objecter ms_handle_connect 0x7f828006e0f0 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.264+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f828c0548e0 con 0x7f829c13f710 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.264+0000 7f8291ffb640 10 client.4459.objecter ms_dispatch 0x7f829c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.264+0000 7f8291ffb640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.264+0000 7f8291ffb640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.264+0000 7f8291ffb640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.264+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f828c050220 con 0x7f829c13f710 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.265+0000 7f8291ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.265+0000 7f8291ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:29.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.496+0000 7f82a336b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.496+0000 7f82a336b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.496+0000 7f82a336b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:29.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.496+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f829c071cc0 con 0x7f829c13f710 2024-03-25T18:24:29.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.520+0000 7f8291ffb640 1 -- 172.21.15.22:0/1183501827 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f828c052320 con 0x7f829c13f710 2024-03-25T18:24:29.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.520+0000 7f8291ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.520+0000 7f8291ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:29.522 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:24:29.522 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:29.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 10 librados: watch_flush enter 2024-03-25T18:24:29.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 10 librados: watch_flush exit 2024-03-25T18:24:29.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:24:29.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:24:29.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 >> v1:172.21.15.33:6800/215350958 conn(0x7f828006e0f0 legacy=0x7f82800705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 10 monclient: shutdown 2024-03-25T18:24:29.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.522+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 >> v1:172.21.15.33:6789/0 conn(0x7f829c13f710 legacy=0x7f829c1e38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.523+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 shutdown_connections 2024-03-25T18:24:29.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.523+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 >> 172.21.15.22:0/1183501827 conn(0x7f829c0b7ea0 msgr2=0x7f829c13b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.523+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 shutdown_connections 2024-03-25T18:24:29.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.523+0000 7f82a336b640 1 -- 172.21.15.22:0/1183501827 wait complete. 2024-03-25T18:24:29.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:29.523+0000 7f82a336b640 1 librados: shutdown 2024-03-25T18:24:29.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:36.343 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:36.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.454+0000 7f252ff0e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:36.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.454+0000 7f252ff0e640 10 monclient: build_initial_monmap 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.454+0000 7f252ff0e640 10 monclient: monmap: 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:36.455748+0000 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:36.455748+0000 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:36.456 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:36.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.455+0000 7f252ff0e640 1 Processor -- start 2024-03-25T18:24:36.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.455+0000 7f252ff0e640 1 -- start start 2024-03-25T18:24:36.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.455+0000 7f252ff0e640 10 monclient: init 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.456+0000 7f252ff0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.456+0000 7f252ff0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.456+0000 7f252ff0e640 10 monclient(hunting): picked mon.noname-a con 0x7f25280a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.456+0000 7f252ff0e640 10 monclient(hunting): picked mon.noname-c con 0x7f252813f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.457+0000 7f252ff0e640 10 monclient(hunting): picked mon.noname-b con 0x7f25281430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.457+0000 7f252ff0e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528142520 con 0x7f25280a98f0 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.457+0000 7f252ff0e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280b28a0 con 0x7f252813f710 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.457+0000 7f252ff0e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280b2a30 con 0x7f25281430c0 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.457+0000 7f252ff0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:36.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.457+0000 7f252ff0e640 10 monclient(hunting): authenticate will time out at 7475.680781s 2024-03-25T18:24:36.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.468+0000 7f252d4a9640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f252813f710 0x7f25281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:41446/0 (socket says 172.21.15.22:41446) 2024-03-25T18:24:36.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.468+0000 7f252d4a9640 1 -- 172.21.15.22:0/126016516 learned_addr learned my addr 172.21.15.22:0/126016516 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:36.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327889086 0 0) 0x7f25280b28a0 con 0x7f252813f710 2024-03-25T18:24:36.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2514003520 con 0x7f252813f710 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059037495 0 0) 0x7f25280b2a30 con 0x7f25281430c0 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:24:36.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.490+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25280b28a0 con 0x7f25281430c0 2024-03-25T18:24:36.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.498+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556400826 0 0) 0x7f2528142520 con 0x7f25280a98f0 2024-03-25T18:24:36.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.498+0000 7f252cca8640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.498+0000 7f252cca8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.498+0000 7f252cca8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.498+0000 7f252cca8640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:24:36.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.498+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25280b2a30 con 0x7f25280a98f0 2024-03-25T18:24:36.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.507+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3892347252 0 0) 0x7f2514003520 con 0x7f252813f710 2024-03-25T18:24:36.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.507+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 >> v1:172.21.15.22:6790/0 conn(0x7f25281430c0 legacy=0x7f252814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.507+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 >> v1:172.21.15.22:6789/0 conn(0x7f25280a98f0 legacy=0x7f25280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 10 monclient: _finish_hunting 0 2024-03-25T18:24:36.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 1 monclient: found mon.noname-c 2024-03-25T18:24:36.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:36.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:24:36.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25280b44c0 con 0x7f252813f710 2024-03-25T18:24:36.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 10 monclient: _finish_auth 0 2024-03-25T18:24:36.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 10 monclient: _check_auth_tickets 2024-03-25T18:24:36.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:36.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2518001690 con 0x7f252813f710 2024-03-25T18:24:36.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:36.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252cca8640 10 monclient: dump: 2024-03-25T18:24:36.512 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:36.512 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:36.512 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.513 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.513 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.513 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:36.513 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:36.513 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:36.513 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:36.513 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:36.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252ff0e640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:24:36.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.508+0000 7f252ff0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:36.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25180017f0 con 0x7f252813f710 2024-03-25T18:24:36.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252cca8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:36.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252cca8640 1 -- 172.21.15.22:0/126016516 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2518001c00 con 0x7f252813f710 2024-03-25T18:24:36.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252ff0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:36.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252ff0e640 4 set_mon_vals no callback set 2024-03-25T18:24:36.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252cca8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252cca8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252cca8640 10 monclient: dump: 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:36.535 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:36.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252ff0e640 10 monclient: shutdown 2024-03-25T18:24:36.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252ff0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:36.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252ff0e640 1 -- 172.21.15.22:0/126016516 >> v1:172.21.15.33:6789/0 conn(0x7f252813f710 legacy=0x7f25281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252ff0e640 1 -- 172.21.15.22:0/126016516 shutdown_connections 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.533+0000 7f252ff0e640 1 -- 172.21.15.22:0/126016516 >> 172.21.15.22:0/126016516 conn(0x7f25280b7ea0 msgr2=0x7f25280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.534+0000 7f252ff0e640 1 -- 172.21.15.22:0/126016516 shutdown_connections 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.534+0000 7f252ff0e640 1 -- 172.21.15.22:0/126016516 wait complete. 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.534+0000 7f252ff0e640 10 monclient: build_initial_monmap 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.534+0000 7f252ff0e640 10 monclient: monmap: 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:36.538 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 librados: starting msgr at 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 librados: starting objecter 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 Processor -- start 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 -- start start 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 librados: setting wanted keys 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 librados: calling monclient init 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient: init 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient(hunting): picked mon.noname-c con 0x7f25280a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient(hunting): picked mon.noname-a con 0x7f252813f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient(hunting): picked mon.noname-b con 0x7f25281430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2528138a50 con 0x7f252813f710 2024-03-25T18:24:36.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25280ab9e0 con 0x7f25281430c0 2024-03-25T18:24:36.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25281d7880 con 0x7f25280a98f0 2024-03-25T18:24:36.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:36.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.535+0000 7f252ff0e640 10 monclient(hunting): authenticate will time out at 7475.759400s 2024-03-25T18:24:36.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.544+0000 7f252e4ab640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f25281430c0 0x7f25281e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:41504/0 (socket says 172.21.15.22:41504) 2024-03-25T18:24:36.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.544+0000 7f252e4ab640 1 -- 172.21.15.22:0/3519478949 learned_addr learned my addr 172.21.15.22:0/3519478949 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:36.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.545+0000 7f251e7fc640 10 client.?.objecter ms_handle_connect 0x7f25281430c0 2024-03-25T18:24:36.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.545+0000 7f251e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.545+0000 7f252e4ab640 1 -- 172.21.15.22:0/3519478949 >> v1:172.21.15.33:6789/0 conn(0x7f25281430c0 legacy=0x7f25281e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:24:36.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.545+0000 7f252e4ab640 1 -- 172.21.15.22:0/3519478949 >> v1:172.21.15.33:6789/0 conn(0x7f25281430c0 legacy=0x7f25281e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:24:36.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.545+0000 7f252e4ab640 1 --1- 172.21.15.22:0/3519478949 >> v1:172.21.15.33:6789/0 conn(0x7f25281430c0 0x7f25281e1f10 :-1 s=OPENED pgs=35643 cs=1 l=1).handle_message read tag failed 2024-03-25T18:24:36.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.545+0000 7f252e4ab640 1 --1- 172.21.15.22:0/3519478949 >> v1:172.21.15.33:6789/0 conn(0x7f25281430c0 0x7f25281e1f10 :-1 s=OPENED pgs=35643 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:24:36.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.545+0000 7f251e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.33:6789/0 2024-03-25T18:24:36.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.564+0000 7f251e7fc640 10 client.?.objecter ms_handle_connect 0x7f252813f710 2024-03-25T18:24:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.564+0000 7f251e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.570+0000 7f251e7fc640 10 client.?.objecter ms_handle_connect 0x7f25280a98f0 2024-03-25T18:24:36.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.570+0000 7f251e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.572+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951090311 0 0) 0x7f2528138a50 con 0x7f252813f710 2024-03-25T18:24:36.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.572+0000 7f251e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.572+0000 7f251e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.572+0000 7f251e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.572+0000 7f251e7fc640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:24:36.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.572+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c004eb0 con 0x7f252813f710 2024-03-25T18:24:36.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.580+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324620622 0 0) 0x7f25281d7880 con 0x7f25280a98f0 2024-03-25T18:24:36.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.580+0000 7f251e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.580+0000 7f251e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.580+0000 7f251e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.580+0000 7f251e7fc640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:24:36.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.580+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2528138a50 con 0x7f25280a98f0 2024-03-25T18:24:36.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.585+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2969126614 0 0) 0x7f250c004eb0 con 0x7f252813f710 2024-03-25T18:24:36.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.585+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25281d7880 con 0x7f252813f710 2024-03-25T18:24:36.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.585+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2518001b00 con 0x7f252813f710 2024-03-25T18:24:36.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.585+0000 7f251e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.585+0000 7f251e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:36.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.585+0000 7f251e7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:36.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.585+0000 7f251e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:36.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.595+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2110128249 0 0) 0x7f2528138a50 con 0x7f25280a98f0 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.595+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f250c004eb0 con 0x7f25280a98f0 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.595+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f25240018f0 con 0x7f25280a98f0 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.595+0000 7f251e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.595+0000 7f251e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.595+0000 7f251e7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:36.597 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:36.598 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:36.598 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:36.598 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1816781695 0 0) 0x7f250c004eb0 con 0x7f25280a98f0 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 >> v1:172.21.15.33:6789/0 conn(0x7f25281430c0 legacy=0x7f25281e1f10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 >> v1:172.21.15.22:6789/0 conn(0x7f252813f710 legacy=0x7f25281d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 1 monclient: found mon.c 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25281d84b0 con 0x7f25280a98f0 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:36.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f251e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 5 monclient: authenticate success, global_id 4466 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 10 monclient: _renew_subs 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25281d51f0 con 0x7f25280a98f0 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 10 monclient: _renew_subs 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2528138e60 con 0x7f25280a98f0 2024-03-25T18:24:36.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.607+0000 7f252ff0e640 1 librados: init done 2024-03-25T18:24:36.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.608+0000 7f252ff0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.608+0000 7f252ff0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.608+0000 7f252ff0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:36.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.608+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25281d76e0 con 0x7f25280a98f0 2024-03-25T18:24:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2524003610 con 0x7f25280a98f0 2024-03-25T18:24:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2524003880 con 0x7f25280a98f0 2024-03-25T18:24:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251e7fc640 10 monclient: dump: 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:36.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251ffff640 4 set_mon_vals no callback set 2024-03-25T18:24:36.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.935+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f2524003aa0 con 0x7f25280a98f0 2024-03-25T18:24:36.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.936+0000 7f252d4a9640 10 monclient: get_auth_request con 0x7f250c06ddf0 auth_method 0 2024-03-25T18:24:36.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.936+0000 7f251e7fc640 10 client.4466.objecter ms_handle_connect 0x7f250c06ddf0 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.942+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f25240533e0 con 0x7f25280a98f0 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.942+0000 7f251e7fc640 10 client.4466.objecter ms_dispatch 0x7f2528000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.942+0000 7f251e7fc640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.942+0000 7f251e7fc640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.942+0000 7f251e7fc640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.942+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f252404eef0 con 0x7f25280a98f0 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.942+0000 7f251e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:36.943+0000 7f251e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:37.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.212+0000 7f252ff0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:37.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.212+0000 7f252ff0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:37.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.212+0000 7f252ff0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:24:37.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.212+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25281d7a10 con 0x7f25280a98f0 2024-03-25T18:24:37.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.248+0000 7f251e7fc640 1 -- 172.21.15.22:0/3519478949 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f2524050e20 con 0x7f25280a98f0 2024-03-25T18:24:37.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.248+0000 7f251e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:37.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.248+0000 7f251e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:37.250 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:24:37.250 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 10 librados: watch_flush enter 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 10 librados: watch_flush exit 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 10 client.4466.objecter successfully canceled tick 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 >> v1:172.21.15.33:6800/215350958 conn(0x7f250c06ddf0 legacy=0x7f250c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 10 monclient: shutdown 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:37.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 >> v1:172.21.15.22:6790/0 conn(0x7f25280a98f0 legacy=0x7f2528138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 shutdown_connections 2024-03-25T18:24:37.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 >> 172.21.15.22:0/3519478949 conn(0x7f25280b7ea0 msgr2=0x7f2528074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:37.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 shutdown_connections 2024-03-25T18:24:37.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 1 -- 172.21.15.22:0/3519478949 wait complete. 2024-03-25T18:24:37.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:37.251+0000 7f252ff0e640 1 librados: shutdown 2024-03-25T18:24:37.265 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:38.177 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:05:40.839 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.181 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:05:40.843 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.243 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:05:40.905 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.251 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:05:40.869 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.252 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:05:40.914 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.257 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:05:40.875 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.260 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:05:40.921 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.278 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:05:40.896 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:38.297 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:05:40.959 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:44.071 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:44.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.191+0000 7f3d63a9a640 10 monclient: get_monmap_and_config 2024-03-25T18:24:44.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.191+0000 7f3d63a9a640 10 monclient: build_initial_monmap 2024-03-25T18:24:44.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.191+0000 7f3d63a9a640 10 monclient: monmap: 2024-03-25T18:24:44.192 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:44.192 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:44.192351+0000 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:44.192351+0000 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:44.193 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.191+0000 7f3d63a9a640 1 Processor -- start 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 1 -- start start 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 10 monclient: init 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c06ea40 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5c071800 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 10 monclient(hunting): picked mon.noname-b con 0x7f3d5c071c40 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c06f1f0 con 0x7f3d5c071800 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.192+0000 7f3d63a9a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c070d00 con 0x7f3d5c06ea40 2024-03-25T18:24:44.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.193+0000 7f3d63a9a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c0720b0 con 0x7f3d5c071c40 2024-03-25T18:24:44.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.193+0000 7f3d63a9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:44.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.193+0000 7f3d63a9a640 10 monclient(hunting): authenticate will time out at 7483.416606s 2024-03-25T18:24:44.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.217+0000 7f3d61836640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f3d5c06ea40 0x7f3d5c13ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:41698/0 (socket says 172.21.15.22:41698) 2024-03-25T18:24:44.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.217+0000 7f3d61836640 1 -- 172.21.15.22:0/2195246373 learned_addr learned my addr 172.21.15.22:0/2195246373 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:44.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.228+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120590526 0 0) 0x7f3d5c070d00 con 0x7f3d5c06ea40 2024-03-25T18:24:44.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.228+0000 7f3d60834640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.228+0000 7f3d60834640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.228+0000 7f3d60834640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.228+0000 7f3d60834640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:24:44.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.228+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48003520 con 0x7f3d5c06ea40 2024-03-25T18:24:44.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3748400231 0 0) 0x7f3d48003520 con 0x7f3d5c06ea40 2024-03-25T18:24:44.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 >> v1:172.21.15.22:6790/0 conn(0x7f3d5c071c40 legacy=0x7f3d5c14a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 >> v1:172.21.15.22:6789/0 conn(0x7f3d5c071800 legacy=0x7f3d5c147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 10 monclient: _finish_hunting 0 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 1 monclient: found mon.noname-c 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c072780 con 0x7f3d5c06ea40 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 10 monclient: _finish_auth 0 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 10 monclient: _check_auth_tickets 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3d58001690 con 0x7f3d5c06ea40 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d63a9a640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d63a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d60834640 10 monclient: dump: 2024-03-25T18:24:44.238 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:44.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.236+0000 7f3d63a9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d580017f0 con 0x7f3d5c06ea40 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d60834640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d60834640 1 -- 172.21.15.22:0/2195246373 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3d58001c00 con 0x7f3d5c06ea40 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d63a9a640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d63a9a640 4 set_mon_vals no callback set 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d60834640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d60834640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.247+0000 7f3d60834640 10 monclient: dump: 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:44.251 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.248+0000 7f3d63a9a640 10 monclient: shutdown 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.248+0000 7f3d63a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.248+0000 7f3d63a9a640 1 -- 172.21.15.22:0/2195246373 >> v1:172.21.15.33:6789/0 conn(0x7f3d5c06ea40 legacy=0x7f3d5c13ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.248+0000 7f3d63a9a640 1 -- 172.21.15.22:0/2195246373 shutdown_connections 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.248+0000 7f3d63a9a640 1 -- 172.21.15.22:0/2195246373 >> 172.21.15.22:0/2195246373 conn(0x7f3d5c06b630 msgr2=0x7f3d5c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.249+0000 7f3d63a9a640 1 -- 172.21.15.22:0/2195246373 shutdown_connections 2024-03-25T18:24:44.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.249+0000 7f3d63a9a640 1 -- 172.21.15.22:0/2195246373 wait complete. 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 10 monclient: build_initial_monmap 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 10 monclient: monmap: 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:44.254 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 1 librados: starting msgr at 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 1 librados: starting objecter 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 1 Processor -- start 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 1 -- start start 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 1 librados: setting wanted keys 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 1 librados: calling monclient init 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.250+0000 7f3d63a9a640 10 monclient: init 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5c06ea40 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 10 monclient(hunting): picked mon.noname-b con 0x7f3d5c071800 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c071c40 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c132450 con 0x7f3d5c06ea40 2024-03-25T18:24:44.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c11f4d0 con 0x7f3d5c071800 2024-03-25T18:24:44.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c132e30 con 0x7f3d5c071c40 2024-03-25T18:24:44.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:44.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.251+0000 7f3d63a9a640 10 monclient(hunting): authenticate will time out at 7483.475283s 2024-03-25T18:24:44.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.259+0000 7f3d61035640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f3d5c071800 0x7f3d5c1daf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:41750/0 (socket says 172.21.15.22:41750) 2024-03-25T18:24:44.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.259+0000 7f3d61035640 1 -- 172.21.15.22:0/3080589263 learned_addr learned my addr 172.21.15.22:0/3080589263 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:44.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.262+0000 7f3d527fc640 10 client.?.objecter ms_handle_connect 0x7f3d5c071800 2024-03-25T18:24:44.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.262+0000 7f3d527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.274+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019106693 0 0) 0x7f3d5c11f4d0 con 0x7f3d5c071800 2024-03-25T18:24:44.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.274+0000 7f3d527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.274+0000 7f3d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.274+0000 7f3d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.274+0000 7f3d527fc640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:24:44.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.274+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d40004eb0 con 0x7f3d5c071800 2024-03-25T18:24:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.283+0000 7f3d527fc640 10 client.?.objecter ms_handle_connect 0x7f3d5c071c40 2024-03-25T18:24:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.283+0000 7f3d527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.290+0000 7f3d527fc640 10 client.?.objecter ms_handle_connect 0x7f3d5c06ea40 2024-03-25T18:24:44.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.290+0000 7f3d527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.292+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 48161782 0 0) 0x7f3d40004eb0 con 0x7f3d5c071800 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.292+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d5c11f4d0 con 0x7f3d5c071800 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.292+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3d4c001a30 con 0x7f3d5c071800 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.292+0000 7f3d527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.292+0000 7f3d527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.292+0000 7f3d527fc640 10 monclient(hunting): dump: 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.294 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:44.295 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:44.295 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:44.295 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:44.295 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.292+0000 7f3d527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.298+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624607672 0 0) 0x7f3d5c132e30 con 0x7f3d5c071c40 2024-03-25T18:24:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.298+0000 7f3d527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.298+0000 7f3d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.298+0000 7f3d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.298+0000 7f3d527fc640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:24:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.298+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d40004eb0 con 0x7f3d5c071c40 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711512078 0 0) 0x7f3d5c132450 con 0x7f3d5c06ea40 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c132e30 con 0x7f3d5c06ea40 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2696681036 0 0) 0x7f3d5c11f4d0 con 0x7f3d5c071800 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 >> v1:172.21.15.22:6790/0 conn(0x7f3d5c071c40 legacy=0x7f3d5c1db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 >> v1:172.21.15.22:6789/0 conn(0x7f3d5c06ea40 legacy=0x7f3d5c11bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 1 monclient: found mon.b 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:44.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c1330a0 con 0x7f3d5c071800 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 5 monclient: authenticate success, global_id 4468 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 10 monclient: _renew_subs 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c1dbd80 con 0x7f3d5c071800 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 10 monclient: _renew_subs 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c11c5b0 con 0x7f3d5c071800 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.306+0000 7f3d63a9a640 1 librados: init done 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.307+0000 7f3d63a9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.307+0000 7f3d63a9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.307+0000 7f3d63a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:44.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.307+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5c074d80 con 0x7f3d5c071800 2024-03-25T18:24:44.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.595+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d4c001cb0 con 0x7f3d5c071800 2024-03-25T18:24:44.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.595+0000 7f3d527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.595+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3d4c0038c0 con 0x7f3d5c071800 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.595+0000 7f3d527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.595+0000 7f3d527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.595+0000 7f3d527fc640 10 monclient: dump: 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:44.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.595+0000 7f3d53fff640 4 set_mon_vals no callback set 2024-03-25T18:24:44.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.596+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f3d4c003aa0 con 0x7f3d5c071800 2024-03-25T18:24:44.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.596+0000 7f3d61836640 10 monclient: get_auth_request con 0x7f3d4006de80 auth_method 0 2024-03-25T18:24:44.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.597+0000 7f3d527fc640 10 client.4468.objecter ms_handle_connect 0x7f3d4006de80 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.602+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f3d4c0547b0 con 0x7f3d5c071800 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.602+0000 7f3d527fc640 10 client.4468.objecter ms_dispatch 0x7f3d5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.602+0000 7f3d527fc640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.602+0000 7f3d527fc640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.602+0000 7f3d527fc640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.603+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d4c0500f0 con 0x7f3d5c071800 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.603+0000 7f3d527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.603+0000 7f3d527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:44.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.857+0000 7f3d63a9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:44.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.857+0000 7f3d63a9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:44.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.857+0000 7f3d63a9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:44.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.857+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d5c11b6f0 con 0x7f3d5c071800 2024-03-25T18:24:44.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.964+0000 7f3d527fc640 1 -- 172.21.15.22:0/3080589263 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f3d4c0521f0 con 0x7f3d5c071800 2024-03-25T18:24:44.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.964+0000 7f3d527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:44.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.964+0000 7f3d527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:44.966 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:24:44.966 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 10 librados: watch_flush enter 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 10 librados: watch_flush exit 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 10 client.4468.objecter successfully canceled tick 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 >> v1:172.21.15.33:6800/215350958 conn(0x7f3d4006de80 legacy=0x7f3d40070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 10 monclient: shutdown 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 >> v1:172.21.15.33:6789/0 conn(0x7f3d5c071800 legacy=0x7f3d5c1daf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 shutdown_connections 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 >> 172.21.15.22:0/3080589263 conn(0x7f3d5c06b630 msgr2=0x7f3d5c0af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 shutdown_connections 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 1 -- 172.21.15.22:0/3080589263 wait complete. 2024-03-25T18:24:44.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:44.967+0000 7f3d63a9a640 1 librados: shutdown 2024-03-25T18:24:44.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:51.787 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:51.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.904+0000 7fd4ef99d640 10 monclient: get_monmap_and_config 2024-03-25T18:24:51.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.904+0000 7fd4ef99d640 10 monclient: build_initial_monmap 2024-03-25T18:24:51.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.904+0000 7fd4ef99d640 10 monclient: monmap: 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:51.905334+0000 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:51.905334+0000 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:51.906 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.904+0000 7fd4ef99d640 1 Processor -- start 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.905+0000 7fd4ef99d640 1 -- start start 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.905+0000 7fd4ef99d640 10 monclient: init 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.905+0000 7fd4ef99d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.905+0000 7fd4ef99d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.905+0000 7fd4ef99d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4e80a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.906+0000 7fd4ef99d640 10 monclient(hunting): picked mon.noname-b con 0x7fd4e813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:51.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.906+0000 7fd4ef99d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4e81430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:51.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.906+0000 7fd4ef99d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e8142520 con 0x7fd4e81430c0 2024-03-25T18:24:51.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.906+0000 7fd4ef99d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e80b28a0 con 0x7fd4e80a98f0 2024-03-25T18:24:51.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.906+0000 7fd4ef99d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e80b2a30 con 0x7fd4e813f710 2024-03-25T18:24:51.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.906+0000 7fd4ef99d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:51.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.906+0000 7fd4ef99d640 10 monclient(hunting): authenticate will time out at 7491.129812s 2024-03-25T18:24:51.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.911+0000 7fd4ed739640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fd4e80a98f0 0x7fd4e80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:46296/0 (socket says 172.21.15.22:46296) 2024-03-25T18:24:51.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.911+0000 7fd4ed739640 1 -- 172.21.15.22:0/3323905304 learned_addr learned my addr 172.21.15.22:0/3323905304 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:51.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.931+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90152620 0 0) 0x7fd4e80b28a0 con 0x7fd4e80a98f0 2024-03-25T18:24:51.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.931+0000 7fd4dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.931+0000 7fd4dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.931+0000 7fd4dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.931+0000 7fd4dffff640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:24:51.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.931+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d4003520 con 0x7fd4e80a98f0 2024-03-25T18:24:51.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231794151 0 0) 0x7fd4e8142520 con 0x7fd4e81430c0 2024-03-25T18:24:51.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:24:51.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e80b28a0 con 0x7fd4e81430c0 2024-03-25T18:24:51.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527239784 0 0) 0x7fd4e80b2a30 con 0x7fd4e813f710 2024-03-25T18:24:51.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:24:51.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.941+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e8142520 con 0x7fd4e813f710 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2097932526 0 0) 0x7fd4e80b28a0 con 0x7fd4e81430c0 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 >> v1:172.21.15.22:6790/0 conn(0x7fd4e813f710 legacy=0x7fd4e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 >> v1:172.21.15.33:6789/0 conn(0x7fd4e80a98f0 legacy=0x7fd4e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 1 monclient: found mon.noname-a 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80b44c0 con 0x7fd4e81430c0 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4ef99d640 5 monclient: authenticate success, global_id 4470 2024-03-25T18:24:51.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.953+0000 7fd4ef99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.957+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd4e00019b0 con 0x7fd4e81430c0 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.957+0000 7fd4dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.957+0000 7fd4dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.957+0000 7fd4dffff640 10 monclient: dump: 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:51.959 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:51.960 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:51.960 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:51.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.957+0000 7fd4ef99d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4e0001c00 con 0x7fd4e81430c0 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4dffff640 1 -- 172.21.15.22:0/3323905304 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd4e0001f90 con 0x7fd4e81430c0 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4ef99d640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4ef99d640 4 set_mon_vals no callback set 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.970+0000 7fd4dffff640 10 monclient: dump: 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:51.972 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:51.973 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:51.973 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:51.973 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:51.973 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:51.973 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:51.973 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:51.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.971+0000 7fd4ef99d640 10 monclient: shutdown 2024-03-25T18:24:51.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.971+0000 7fd4ef99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:51.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.971+0000 7fd4ef99d640 1 -- 172.21.15.22:0/3323905304 >> v1:172.21.15.22:6789/0 conn(0x7fd4e81430c0 legacy=0x7fd4e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.971+0000 7fd4ef99d640 1 -- 172.21.15.22:0/3323905304 shutdown_connections 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.971+0000 7fd4ef99d640 1 -- 172.21.15.22:0/3323905304 >> 172.21.15.22:0/3323905304 conn(0x7fd4e80b7ea0 msgr2=0x7fd4e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.971+0000 7fd4ef99d640 1 -- 172.21.15.22:0/3323905304 shutdown_connections 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.971+0000 7fd4ef99d640 1 -- 172.21.15.22:0/3323905304 wait complete. 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.972+0000 7fd4ef99d640 10 monclient: build_initial_monmap 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.972+0000 7fd4ef99d640 10 monclient: monmap: 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.972+0000 7fd4ef99d640 1 librados: starting msgr at 2024-03-25T18:24:51.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.972+0000 7fd4ef99d640 1 librados: starting objecter 2024-03-25T18:24:51.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 1 Processor -- start 2024-03-25T18:24:51.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 1 -- start start 2024-03-25T18:24:51.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 1 librados: setting wanted keys 2024-03-25T18:24:51.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 1 librados: calling monclient init 2024-03-25T18:24:51.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 10 monclient: init 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 10 monclient(hunting): picked mon.noname-c con 0x7fd4e80a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 10 monclient(hunting): picked mon.noname-a con 0x7fd4e813f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.973+0000 7fd4ef99d640 10 monclient(hunting): picked mon.noname-b con 0x7fd4e81430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.974+0000 7fd4ef99d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e81db790 con 0x7fd4e813f710 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.974+0000 7fd4ef99d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e8137f40 con 0x7fd4e81430c0 2024-03-25T18:24:51.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.974+0000 7fd4ef99d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e81da880 con 0x7fd4e80a98f0 2024-03-25T18:24:51.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.974+0000 7fd4ef99d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:51.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.974+0000 7fd4ef99d640 10 monclient(hunting): authenticate will time out at 7491.197642s 2024-03-25T18:24:51.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.986+0000 7fd4edf3a640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fd4e81430c0 0x7fd4e8137830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:46360/0 (socket says 172.21.15.22:46360) 2024-03-25T18:24:51.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.986+0000 7fd4edf3a640 1 -- 172.21.15.22:0/2474499409 learned_addr learned my addr 172.21.15.22:0/2474499409 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:51.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.987+0000 7fd4ddffb640 10 client.?.objecter ms_handle_connect 0x7fd4e81430c0 2024-03-25T18:24:51.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.987+0000 7fd4ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:51.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.997+0000 7fd4ddffb640 10 client.?.objecter ms_handle_connect 0x7fd4e80a98f0 2024-03-25T18:24:51.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.997+0000 7fd4ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:52.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.999+0000 7fd4ddffb640 10 client.?.objecter ms_handle_connect 0x7fd4e813f710 2024-03-25T18:24:52.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:51.999+0000 7fd4ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:52.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.000+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742135756 0 0) 0x7fd4e8137f40 con 0x7fd4e81430c0 2024-03-25T18:24:52.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.000+0000 7fd4ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.000+0000 7fd4ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.000+0000 7fd4ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.000+0000 7fd4ddffb640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:24:52.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.000+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4cc004eb0 con 0x7fd4e81430c0 2024-03-25T18:24:52.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.009+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015495122 0 0) 0x7fd4e81da880 con 0x7fd4e80a98f0 2024-03-25T18:24:52.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.009+0000 7fd4ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.009+0000 7fd4ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.009+0000 7fd4ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.009+0000 7fd4ddffb640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:24:52.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.009+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e8137f40 con 0x7fd4e80a98f0 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.017+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 438448491 0 0) 0x7fd4cc004eb0 con 0x7fd4e81430c0 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.017+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4e81da880 con 0x7fd4e81430c0 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.017+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd4e0001f50 con 0x7fd4e81430c0 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.017+0000 7fd4ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.017+0000 7fd4ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.017+0000 7fd4ddffb640 10 monclient(hunting): dump: 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:52.019 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:52.020 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:52.020 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:52.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.017+0000 7fd4ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.020+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939561412 0 0) 0x7fd4e81db790 con 0x7fd4e813f710 2024-03-25T18:24:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.020+0000 7fd4ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.020+0000 7fd4ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.020+0000 7fd4ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.020+0000 7fd4ddffb640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:24:52.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.020+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4cc004eb0 con 0x7fd4e813f710 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2860422446 0 0) 0x7fd4e81da880 con 0x7fd4e81430c0 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 >> v1:172.21.15.22:6790/0 conn(0x7fd4e80a98f0 legacy=0x7fd4e8138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 >> v1:172.21.15.22:6789/0 conn(0x7fd4e813f710 legacy=0x7fd4e81db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 1 monclient: found mon.b 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e81dab60 con 0x7fd4e81430c0 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:24:52.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 10 monclient: _renew_subs 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e81e2bc0 con 0x7fd4e81430c0 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 10 monclient: _renew_subs 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e81e2f30 con 0x7fd4e81430c0 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.021+0000 7fd4ef99d640 1 librados: init done 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.022+0000 7fd4ef99d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:52.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.022+0000 7fd4ef99d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.022+0000 7fd4ef99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.022+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e80627b0 con 0x7fd4e81430c0 2024-03-25T18:24:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.024+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4e0003c20 con 0x7fd4e81430c0 2024-03-25T18:24:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.024+0000 7fd4ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.024+0000 7fd4df7fe640 4 set_mon_vals no callback set 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.297+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd4e00015b0 con 0x7fd4e81430c0 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.297+0000 7fd4ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.297+0000 7fd4ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.297+0000 7fd4ddffb640 10 monclient: dump: 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:52.299 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:52.300 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:52.300 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:52.300 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:52.300 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:52.300 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:52.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.298+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fd4e00040b0 con 0x7fd4e81430c0 2024-03-25T18:24:52.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.299+0000 7fd4ed739640 10 monclient: get_auth_request con 0x7fd4cc06de00 auth_method 0 2024-03-25T18:24:52.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.299+0000 7fd4ddffb640 10 client.4474.objecter ms_handle_connect 0x7fd4cc06de00 2024-03-25T18:24:52.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.301+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fd4e0054cc0 con 0x7fd4e81430c0 2024-03-25T18:24:52.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.301+0000 7fd4ddffb640 10 client.4474.objecter ms_dispatch 0x7fd4e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:52.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.301+0000 7fd4ddffb640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:52.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.301+0000 7fd4ddffb640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:52.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.301+0000 7fd4ddffb640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-25T18:24:52.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.305+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4e0050600 con 0x7fd4e81430c0 2024-03-25T18:24:52.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.305+0000 7fd4ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:52.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.305+0000 7fd4ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:52.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.529+0000 7fd4ef99d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.529+0000 7fd4ef99d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.529+0000 7fd4ef99d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:24:52.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.529+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4e81dacf0 con 0x7fd4e81430c0 2024-03-25T18:24:52.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.561+0000 7fd4ddffb640 1 -- 172.21.15.22:0/2474499409 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fd4e0052700 con 0x7fd4e81430c0 2024-03-25T18:24:52.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.561+0000 7fd4ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.561+0000 7fd4ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:52.563 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:24:52.563 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:52.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 10 librados: watch_flush enter 2024-03-25T18:24:52.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 10 librados: watch_flush exit 2024-03-25T18:24:52.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-25T18:24:52.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 10 client.4474.objecter successfully canceled tick 2024-03-25T18:24:52.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 >> v1:172.21.15.33:6800/215350958 conn(0x7fd4cc06de00 legacy=0x7fd4cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 10 monclient: shutdown 2024-03-25T18:24:52.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:52.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 >> v1:172.21.15.33:6789/0 conn(0x7fd4e81430c0 legacy=0x7fd4e8137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 shutdown_connections 2024-03-25T18:24:52.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 >> 172.21.15.22:0/2474499409 conn(0x7fd4e80b7ea0 msgr2=0x7fd4e813d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:52.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 shutdown_connections 2024-03-25T18:24:52.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 1 -- 172.21.15.22:0/2474499409 wait complete. 2024-03-25T18:24:52.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:52.564+0000 7fd4ef99d640 1 librados: shutdown 2024-03-25T18:24:52.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:58.234 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:00.896 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.238 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:00.900 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.276 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:00.938 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.308 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:00.970 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.314 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:00.975 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.332 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:00.950 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.348 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:00.966 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.357 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:01.019 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:58.369 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:00.988 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.382 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.497+0000 7fcc2f6c3640 10 monclient: get_monmap_and_config 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.497+0000 7fcc2f6c3640 10 monclient: build_initial_monmap 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.497+0000 7fcc2f6c3640 10 monclient: monmap: 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:24:59.498998+0000 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:24:59.498998+0000 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:59.499 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:24:59.500 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:24:59.500 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.498+0000 7fcc2f6c3640 1 Processor -- start 2024-03-25T18:24:59.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.498+0000 7fcc2f6c3640 1 -- start start 2024-03-25T18:24:59.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.498+0000 7fcc2f6c3640 10 monclient: init 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 10 monclient(hunting): picked mon.noname-c con 0x7fcc280a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 10 monclient(hunting): picked mon.noname-b con 0x7fcc2813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 10 monclient(hunting): picked mon.noname-a con 0x7fcc281430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc28142520 con 0x7fcc281430c0 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280b28a0 con 0x7fcc280a98f0 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc280b2a30 con 0x7fcc2813f710 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:59.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.499+0000 7fcc2f6c3640 10 monclient(hunting): authenticate will time out at 7498.723434s 2024-03-25T18:24:59.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.513+0000 7fcc2dc60640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fcc281430c0 0x7fcc2814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46106/0 (socket says 172.21.15.22:46106) 2024-03-25T18:24:59.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.513+0000 7fcc2dc60640 1 -- 172.21.15.22:0/1305548144 learned_addr learned my addr 172.21.15.22:0/1305548144 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:59.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97919445 0 0) 0x7fcc280b2a30 con 0x7fcc2813f710 2024-03-25T18:24:59.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:24:59.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc14003520 con 0x7fcc2813f710 2024-03-25T18:24:59.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323010715 0 0) 0x7fcc28142520 con 0x7fcc281430c0 2024-03-25T18:24:59.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:24:59.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.527+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc280b2a30 con 0x7fcc281430c0 2024-03-25T18:24:59.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.531+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400522272 0 0) 0x7fcc280b28a0 con 0x7fcc280a98f0 2024-03-25T18:24:59.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.531+0000 7fcc1ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.531+0000 7fcc1ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.531+0000 7fcc1ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.531+0000 7fcc1ffff640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:24:59.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.531+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28142520 con 0x7fcc280a98f0 2024-03-25T18:24:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1575153713 0 0) 0x7fcc280b2a30 con 0x7fcc281430c0 2024-03-25T18:24:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 >> v1:172.21.15.22:6790/0 conn(0x7fcc2813f710 legacy=0x7fcc281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 >> v1:172.21.15.33:6789/0 conn(0x7fcc280a98f0 legacy=0x7fcc280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 1 monclient: found mon.noname-a 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280b44c0 con 0x7fcc281430c0 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcc200019f0 con 0x7fcc281430c0 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc1ffff640 10 monclient: dump: 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:59.538 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc2f6c3640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:24:59.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.536+0000 7fcc2f6c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:59.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc20001b50 con 0x7fcc281430c0 2024-03-25T18:24:59.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc1ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:59.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc1ffff640 1 -- 172.21.15.22:0/1305548144 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcc20001f60 con 0x7fcc281430c0 2024-03-25T18:24:59.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc2f6c3640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:59.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc2f6c3640 4 set_mon_vals no callback set 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc1ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc1ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.545+0000 7fcc1ffff640 10 monclient: dump: 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:59.547 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.546+0000 7fcc2f6c3640 10 monclient: shutdown 2024-03-25T18:24:59.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.546+0000 7fcc2f6c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:59.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.546+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/1305548144 >> v1:172.21.15.22:6789/0 conn(0x7fcc281430c0 legacy=0x7fcc2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.546+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/1305548144 shutdown_connections 2024-03-25T18:24:59.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.546+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/1305548144 >> 172.21.15.22:0/1305548144 conn(0x7fcc280b7ea0 msgr2=0x7fcc280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:59.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.546+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/1305548144 shutdown_connections 2024-03-25T18:24:59.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.546+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/1305548144 wait complete. 2024-03-25T18:24:59.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.547+0000 7fcc2f6c3640 10 monclient: build_initial_monmap 2024-03-25T18:24:59.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.547+0000 7fcc2f6c3640 10 monclient: monmap: 2024-03-25T18:24:59.549 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:24:59.549 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:59.549 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.547+0000 7fcc2f6c3640 1 librados: starting msgr at 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.547+0000 7fcc2f6c3640 1 librados: starting objecter 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 1 Processor -- start 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 1 -- start start 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 1 librados: setting wanted keys 2024-03-25T18:24:59.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 1 librados: calling monclient init 2024-03-25T18:24:59.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 10 monclient: init 2024-03-25T18:24:59.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:59.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:59.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 10 monclient(hunting): picked mon.noname-b con 0x7fcc280a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:24:59.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 10 monclient(hunting): picked mon.noname-a con 0x7fcc2813f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:24:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 10 monclient(hunting): picked mon.noname-c con 0x7fcc281430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:24:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.548+0000 7fcc2f6c3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc28139920 con 0x7fcc2813f710 2024-03-25T18:24:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.549+0000 7fcc2f6c3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc281db380 con 0x7fcc280a98f0 2024-03-25T18:24:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.549+0000 7fcc2f6c3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc281db620 con 0x7fcc281430c0 2024-03-25T18:24:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.549+0000 7fcc2f6c3640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:59.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.549+0000 7fcc2f6c3640 10 monclient(hunting): authenticate will time out at 7498.772612s 2024-03-25T18:24:59.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.560+0000 7fcc2cc5e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fcc2813f710 0x7fcc280b0020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46136/0 (socket says 172.21.15.22:46136) 2024-03-25T18:24:59.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.560+0000 7fcc2cc5e640 1 -- 172.21.15.22:0/3696746361 learned_addr learned my addr 172.21.15.22:0/3696746361 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:24:59.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.561+0000 7fcc1dffb640 10 client.?.objecter ms_handle_connect 0x7fcc2813f710 2024-03-25T18:24:59.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.561+0000 7fcc1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.562+0000 7fcc1dffb640 10 client.?.objecter ms_handle_connect 0x7fcc281430c0 2024-03-25T18:24:59.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.562+0000 7fcc1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.569+0000 7fcc1dffb640 10 client.?.objecter ms_handle_connect 0x7fcc280a98f0 2024-03-25T18:24:59.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.569+0000 7fcc1dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.580+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13502514 0 0) 0x7fcc281db380 con 0x7fcc280a98f0 2024-03-25T18:24:59.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.580+0000 7fcc1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.580+0000 7fcc1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.580+0000 7fcc1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.580+0000 7fcc1dffb640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:24:59.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.580+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c004eb0 con 0x7fcc280a98f0 2024-03-25T18:24:59.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.583+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592394484 0 0) 0x7fcc281db620 con 0x7fcc281430c0 2024-03-25T18:24:59.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.583+0000 7fcc1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.583+0000 7fcc1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.583+0000 7fcc1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.583+0000 7fcc1dffb640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:24:59.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.583+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc281db380 con 0x7fcc281430c0 2024-03-25T18:24:59.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.584+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617699981 0 0) 0x7fcc28139920 con 0x7fcc2813f710 2024-03-25T18:24:59.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.584+0000 7fcc1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.584+0000 7fcc1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.584+0000 7fcc1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.584+0000 7fcc1dffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:24:59.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.584+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc281db620 con 0x7fcc2813f710 2024-03-25T18:24:59.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.598+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3605263315 0 0) 0x7fcc0c004eb0 con 0x7fcc280a98f0 2024-03-25T18:24:59.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.598+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc28139920 con 0x7fcc280a98f0 2024-03-25T18:24:59.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.598+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcc240018f0 con 0x7fcc280a98f0 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.598+0000 7fcc1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.598+0000 7fcc1dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.598+0000 7fcc1dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.598+0000 7fcc1dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.599+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563420739 0 0) 0x7fcc281db620 con 0x7fcc2813f710 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.599+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc0c004eb0 con 0x7fcc2813f710 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.599+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcc18001910 con 0x7fcc2813f710 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.599+0000 7fcc1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.599+0000 7fcc1dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.599+0000 7fcc1dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:59.601 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:59.602 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.615+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2443215506 0 0) 0x7fcc281db380 con 0x7fcc281430c0 2024-03-25T18:24:59.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.615+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc281db620 con 0x7fcc281430c0 2024-03-25T18:24:59.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.615+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcc20001e60 con 0x7fcc281430c0 2024-03-25T18:24:59.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.615+0000 7fcc1dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.615+0000 7fcc1dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.615+0000 7fcc1dffb640 10 monclient(hunting): dump: 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:59.617 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4140077243 0 0) 0x7fcc0c004eb0 con 0x7fcc2813f710 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 >> v1:172.21.15.22:6790/0 conn(0x7fcc281430c0 legacy=0x7fcc28137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 >> v1:172.21.15.33:6789/0 conn(0x7fcc280a98f0 legacy=0x7fcc281dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 1 monclient: found mon.a 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:24:59.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc280b0dc0 con 0x7fcc2813f710 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 5 monclient: authenticate success, global_id 4479 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 10 monclient: _renew_subs 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc280b1010 con 0x7fcc2813f710 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 10 monclient: _renew_subs 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc280b0910 con 0x7fcc2813f710 2024-03-25T18:24:59.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.618+0000 7fcc2f6c3640 1 librados: init done 2024-03-25T18:24:59.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.619+0000 7fcc2f6c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.619+0000 7fcc2f6c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.619+0000 7fcc2f6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:24:59.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.619+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc281db7b0 con 0x7fcc2813f710 2024-03-25T18:24:59.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.932+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc18003610 con 0x7fcc2813f710 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.932+0000 7fcc1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.932+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcc180039e0 con 0x7fcc2813f710 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.932+0000 7fcc1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.932+0000 7fcc1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.932+0000 7fcc1dffb640 10 monclient: dump: 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:24:59.934 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:24:59.935 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:24:59.935 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:24:59.935 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:24:59.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.932+0000 7fcc1f7fe640 4 set_mon_vals no callback set 2024-03-25T18:24:59.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.933+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fcc18003c00 con 0x7fcc2813f710 2024-03-25T18:24:59.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.933+0000 7fcc2d45f640 10 monclient: get_auth_request con 0x7fcc0c06e110 auth_method 0 2024-03-25T18:24:59.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.933+0000 7fcc1dffb640 10 client.4479.objecter ms_handle_connect 0x7fcc0c06e110 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fcc18054720 con 0x7fcc2813f710 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 10 client.4479.objecter ms_dispatch 0x7fcc28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc18050230 con 0x7fcc2813f710 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:24:59.938+0000 7fcc1dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:00.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.200+0000 7fcc2f6c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:00.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.200+0000 7fcc2f6c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:00.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.200+0000 7fcc2f6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:00.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.200+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc280b0bc0 con 0x7fcc2813f710 2024-03-25T18:25:00.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.230+0000 7fcc1dffb640 1 -- 172.21.15.22:0/3696746361 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fcc18052160 con 0x7fcc2813f710 2024-03-25T18:25:00.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.230+0000 7fcc1dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:00.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.230+0000 7fcc1dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:00.232 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:00.232 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.231+0000 7fcc2f6c3640 10 librados: watch_flush enter 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.231+0000 7fcc2f6c3640 10 librados: watch_flush exit 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.231+0000 7fcc2f6c3640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.231+0000 7fcc2f6c3640 10 client.4479.objecter successfully canceled tick 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 >> v1:172.21.15.33:6800/215350958 conn(0x7fcc0c06e110 legacy=0x7fcc0c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 10 monclient: shutdown 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 >> v1:172.21.15.22:6789/0 conn(0x7fcc2813f710 legacy=0x7fcc280b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 shutdown_connections 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 >> 172.21.15.22:0/3696746361 conn(0x7fcc280b7ea0 msgr2=0x7fcc28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:00.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 shutdown_connections 2024-03-25T18:25:00.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 1 -- 172.21.15.22:0/3696746361 wait complete. 2024-03-25T18:25:00.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:00.232+0000 7fcc2f6c3640 1 librados: shutdown 2024-03-25T18:25:00.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:07.054 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:07.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.165+0000 7fed8dcf1640 10 monclient: get_monmap_and_config 2024-03-25T18:25:07.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.165+0000 7fed8dcf1640 10 monclient: build_initial_monmap 2024-03-25T18:25:07.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.165+0000 7fed8dcf1640 10 monclient: monmap: 2024-03-25T18:25:07.166 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:07.166 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:07.166 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:25:07.166144+0000 2024-03-25T18:25:07.166 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:25:07.166144+0000 2024-03-25T18:25:07.167 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:07.167 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:07.167 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:07.167 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:25:07.167 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:25:07.167 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:07.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.166+0000 7fed8dcf1640 1 Processor -- start 2024-03-25T18:25:07.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.166+0000 7fed8dcf1640 1 -- start start 2024-03-25T18:25:07.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.166+0000 7fed8dcf1640 10 monclient: init 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.167+0000 7fed8dcf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.167+0000 7fed8dcf1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.167+0000 7fed8dcf1640 10 monclient(hunting): picked mon.noname-b con 0x7fed880a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.167+0000 7fed8dcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fed8813f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.167+0000 7fed8dcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fed881430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.167+0000 7fed8dcf1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed88142520 con 0x7fed881430c0 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.168+0000 7fed8dcf1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed880b28a0 con 0x7fed8813f710 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.168+0000 7fed8dcf1640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed880b2a30 con 0x7fed880a98f0 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.168+0000 7fed8dcf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.168+0000 7fed8dcf1640 10 monclient(hunting): authenticate will time out at 7506.390636s 2024-03-25T18:25:07.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.168+0000 7fed86ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fed8813f710 0x7fed881429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:54172/0 (socket says 172.21.15.22:54172) 2024-03-25T18:25:07.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.168+0000 7fed86ffd640 1 -- 172.21.15.22:0/454967800 learned_addr learned my addr 172.21.15.22:0/454967800 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:07.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.178+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843769179 0 0) 0x7fed880b28a0 con 0x7fed8813f710 2024-03-25T18:25:07.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.178+0000 7fed867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.178+0000 7fed867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.178+0000 7fed867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.178+0000 7fed867fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:25:07.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.178+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed70003520 con 0x7fed8813f710 2024-03-25T18:25:07.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.183+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274259923 0 0) 0x7fed880b2a30 con 0x7fed880a98f0 2024-03-25T18:25:07.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.183+0000 7fed867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.183+0000 7fed867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.183+0000 7fed867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.183+0000 7fed867fc640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:25:07.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.183+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed880b28a0 con 0x7fed880a98f0 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184271564 0 0) 0x7fed70003520 con 0x7fed8813f710 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 >> v1:172.21.15.22:6790/0 conn(0x7fed880a98f0 legacy=0x7fed880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 >> v1:172.21.15.22:6789/0 conn(0x7fed881430c0 legacy=0x7fed8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 1 monclient: found mon.noname-c 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed880b44c0 con 0x7fed8813f710 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fed7c001690 con 0x7fed8813f710 2024-03-25T18:25:07.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed867fc640 10 monclient: dump: 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed8dcf1640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:25:07.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.188+0000 7fed8dcf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed7c0017f0 con 0x7fed8813f710 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed867fc640 1 -- 172.21.15.22:0/454967800 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fed7c001c00 con 0x7fed8813f710 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed8dcf1640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed8dcf1640 4 set_mon_vals no callback set 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.202+0000 7fed867fc640 10 monclient: dump: 2024-03-25T18:25:07.203 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:07.204 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.203+0000 7fed8dcf1640 10 monclient: shutdown 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.203+0000 7fed8dcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.203+0000 7fed8dcf1640 1 -- 172.21.15.22:0/454967800 >> v1:172.21.15.33:6789/0 conn(0x7fed8813f710 legacy=0x7fed881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.203+0000 7fed8dcf1640 1 -- 172.21.15.22:0/454967800 shutdown_connections 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.203+0000 7fed8dcf1640 1 -- 172.21.15.22:0/454967800 >> 172.21.15.22:0/454967800 conn(0x7fed880b7ea0 msgr2=0x7fed880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.203+0000 7fed8dcf1640 1 -- 172.21.15.22:0/454967800 shutdown_connections 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.203+0000 7fed8dcf1640 1 -- 172.21.15.22:0/454967800 wait complete. 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.204+0000 7fed8dcf1640 10 monclient: build_initial_monmap 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.204+0000 7fed8dcf1640 10 monclient: monmap: 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:07.206 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.204+0000 7fed8dcf1640 1 librados: starting msgr at 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.204+0000 7fed8dcf1640 1 librados: starting objecter 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.204+0000 7fed8dcf1640 1 Processor -- start 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 1 -- start start 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 1 librados: setting wanted keys 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 1 librados: calling monclient init 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient: init 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:07.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient(hunting): picked mon.noname-a con 0x7fed880a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient(hunting): picked mon.noname-c con 0x7fed8813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient(hunting): picked mon.noname-b con 0x7fed881430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8813b2b0 con 0x7fed880a98f0 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed8813b500 con 0x7fed881430c0 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed880b6f40 con 0x7fed8813f710 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.205+0000 7fed8dcf1640 10 monclient(hunting): authenticate will time out at 7506.428217s 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.208+0000 7fed877fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fed880a98f0 0x7fed88139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44084/0 (socket says 172.21.15.22:44084) 2024-03-25T18:25:07.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.208+0000 7fed877fe640 1 -- 172.21.15.22:0/659100408 learned_addr learned my addr 172.21.15.22:0/659100408 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:07.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.210+0000 7fed6ffff640 10 client.?.objecter ms_handle_connect 0x7fed880a98f0 2024-03-25T18:25:07.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.210+0000 7fed6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.220+0000 7fed6ffff640 10 client.?.objecter ms_handle_connect 0x7fed881430c0 2024-03-25T18:25:07.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.220+0000 7fed6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.221+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42528619 0 0) 0x7fed8813b2b0 con 0x7fed880a98f0 2024-03-25T18:25:07.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.221+0000 7fed6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.221+0000 7fed6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.221+0000 7fed6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.221+0000 7fed6ffff640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:25:07.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.221+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed74004eb0 con 0x7fed880a98f0 2024-03-25T18:25:07.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.222+0000 7fed6ffff640 10 client.?.objecter ms_handle_connect 0x7fed8813f710 2024-03-25T18:25:07.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.222+0000 7fed6ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:07.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.226+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530438617 0 0) 0x7fed8813b500 con 0x7fed881430c0 2024-03-25T18:25:07.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.226+0000 7fed6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.226+0000 7fed6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.226+0000 7fed6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.226+0000 7fed6ffff640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:25:07.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.226+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8813b2b0 con 0x7fed881430c0 2024-03-25T18:25:07.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.233+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458340931 0 0) 0x7fed880b6f40 con 0x7fed8813f710 2024-03-25T18:25:07.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.233+0000 7fed6ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:07.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.233+0000 7fed6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:07.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.233+0000 7fed6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:07.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.233+0000 7fed6ffff640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:25:07.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.233+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8813b500 con 0x7fed8813f710 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.240+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1473372723 0 0) 0x7fed8813b2b0 con 0x7fed881430c0 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.240+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed880b6f40 con 0x7fed881430c0 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.240+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fed80001a70 con 0x7fed881430c0 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.240+0000 7fed6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.240+0000 7fed6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.240+0000 7fed6ffff640 10 monclient(hunting): dump: 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:07.241 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.240+0000 7fed6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.241+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3002693714 0 0) 0x7fed74004eb0 con 0x7fed880a98f0 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.241+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed8813b2b0 con 0x7fed880a98f0 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.241+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fed78001a10 con 0x7fed880a98f0 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.241+0000 7fed6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.241+0000 7fed6ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.241+0000 7fed6ffff640 10 monclient(hunting): dump: 2024-03-25T18:25:07.242 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:07.243 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:07.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1516270668 0 0) 0x7fed8813b2b0 con 0x7fed880a98f0 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 >> v1:172.21.15.22:6790/0 conn(0x7fed8813f710 legacy=0x7fed881df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 >> v1:172.21.15.33:6789/0 conn(0x7fed881430c0 legacy=0x7fed881e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 1 monclient: found mon.a 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8813a430 con 0x7fed880a98f0 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 5 monclient: authenticate success, global_id 4482 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 10 monclient: _renew_subs 2024-03-25T18:25:07.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8812dd10 con 0x7fed880a98f0 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 10 monclient: _renew_subs 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed880b7b40 con 0x7fed880a98f0 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.251+0000 7fed8dcf1640 1 librados: init done 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.252+0000 7fed8dcf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.252+0000 7fed8dcf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.252+0000 7fed8dcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:07.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.252+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed880627b0 con 0x7fed880a98f0 2024-03-25T18:25:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.574+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed78001c50 con 0x7fed880a98f0 2024-03-25T18:25:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.574+0000 7fed6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.574+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fed78003a60 con 0x7fed880a98f0 2024-03-25T18:25:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.574+0000 7fed6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.574+0000 7fed6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.574+0000 7fed6ffff640 10 monclient: dump: 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.574+0000 7fed85ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:07.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.575+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fed78003c80 con 0x7fed880a98f0 2024-03-25T18:25:07.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.575+0000 7fed86ffd640 10 monclient: get_auth_request con 0x7fed7406df70 auth_method 0 2024-03-25T18:25:07.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.576+0000 7fed6ffff640 10 client.4482.objecter ms_handle_connect 0x7fed7406df70 2024-03-25T18:25:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fed780548b0 con 0x7fed880a98f0 2024-03-25T18:25:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 10 client.4482.objecter ms_dispatch 0x7fed88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-25T18:25:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed780503c0 con 0x7fed880a98f0 2024-03-25T18:25:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.582+0000 7fed6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:07.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.836+0000 7fed8dcf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:07.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.836+0000 7fed8dcf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:07.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.836+0000 7fed8dcf1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:07.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.836+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed881434a0 con 0x7fed880a98f0 2024-03-25T18:25:07.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.860+0000 7fed6ffff640 1 -- 172.21.15.22:0/659100408 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fed780522f0 con 0x7fed880a98f0 2024-03-25T18:25:07.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.860+0000 7fed6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:07.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.860+0000 7fed6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:07.861 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:07.861 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 10 librados: watch_flush enter 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 10 librados: watch_flush exit 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 10 client.4482.objecter successfully canceled tick 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 >> v1:172.21.15.33:6800/215350958 conn(0x7fed7406df70 legacy=0x7fed74070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 10 monclient: shutdown 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 >> v1:172.21.15.22:6789/0 conn(0x7fed880a98f0 legacy=0x7fed88139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 shutdown_connections 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 >> 172.21.15.22:0/659100408 conn(0x7fed880b7ea0 msgr2=0x7fed88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:07.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 shutdown_connections 2024-03-25T18:25:07.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 1 -- 172.21.15.22:0/659100408 wait complete. 2024-03-25T18:25:07.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:07.863+0000 7fed8dcf1640 1 librados: shutdown 2024-03-25T18:25:07.874 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:14.681 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.794+0000 7f3f7684a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.794+0000 7f3f7684a640 10 monclient: build_initial_monmap 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.794+0000 7f3f7684a640 10 monclient: monmap: 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:25:14.794930+0000 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:25:14.794930+0000 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:14.795 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:14.796 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:25:14.796 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:25:14.796 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:14.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.795+0000 7f3f7684a640 1 Processor -- start 2024-03-25T18:25:14.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.795+0000 7f3f7684a640 1 -- start start 2024-03-25T18:25:14.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.795+0000 7f3f7684a640 10 monclient: init 2024-03-25T18:25:14.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.796+0000 7f3f7684a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:14.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.796+0000 7f3f7684a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:14.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.796+0000 7f3f7684a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f700a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:14.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.796+0000 7f3f7684a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7013f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:14.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.796+0000 7f3f7684a640 10 monclient(hunting): picked mon.noname-b con 0x7f3f701430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:14.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.796+0000 7f3f7684a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f70142520 con 0x7f3f700a98f0 2024-03-25T18:25:14.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.796+0000 7f3f7684a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700b28a0 con 0x7f3f7013f710 2024-03-25T18:25:14.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.797+0000 7f3f7684a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700b2a30 con 0x7f3f701430c0 2024-03-25T18:25:14.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.797+0000 7f3f7684a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:14.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.797+0000 7f3f7684a640 10 monclient(hunting): authenticate will time out at 7514.019610s 2024-03-25T18:25:14.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.803+0000 7f3f6f7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f3f7013f710 0x7f3f701429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:55100/0 (socket says 172.21.15.22:55100) 2024-03-25T18:25:14.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.803+0000 7f3f6f7fe640 1 -- 172.21.15.22:0/2678949246 learned_addr learned my addr 172.21.15.22:0/2678949246 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:14.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.815+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384143735 0 0) 0x7f3f700b28a0 con 0x7f3f7013f710 2024-03-25T18:25:14.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.815+0000 7f3f6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.815+0000 7f3f6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.815+0000 7f3f6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.815+0000 7f3f6effd640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:25:14.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.815+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f58003520 con 0x7f3f7013f710 2024-03-25T18:25:14.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245289398 0 0) 0x7f3f700b2a30 con 0x7f3f701430c0 2024-03-25T18:25:14.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:25:14.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f700b28a0 con 0x7f3f701430c0 2024-03-25T18:25:14.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005081336 0 0) 0x7f3f70142520 con 0x7f3f700a98f0 2024-03-25T18:25:14.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:25:14.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.823+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f700b2a30 con 0x7f3f700a98f0 2024-03-25T18:25:14.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2366690851 0 0) 0x7f3f58003520 con 0x7f3f7013f710 2024-03-25T18:25:14.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 >> v1:172.21.15.22:6790/0 conn(0x7f3f701430c0 legacy=0x7f3f7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 >> v1:172.21.15.22:6789/0 conn(0x7f3f700a98f0 legacy=0x7f3f700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 1 monclient: found mon.noname-c 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f700b44c0 con 0x7f3f7013f710 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3f64001690 con 0x7f3f7013f710 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f6effd640 10 monclient: dump: 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:14.828 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.829 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:14.829 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:14.829 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:14.829 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:14.829 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:14.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f7684a640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:25:14.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.827+0000 7f3f7684a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:14.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f640017f0 con 0x7f3f7013f710 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f6effd640 1 -- 172.21.15.22:0/2678949246 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3f64001c00 con 0x7f3f7013f710 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f7684a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f7684a640 4 set_mon_vals no callback set 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.835+0000 7f3f6effd640 10 monclient: dump: 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:14.836 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:14.837 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:14.837 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:14.837 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.836+0000 7f3f7684a640 10 monclient: shutdown 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.836+0000 7f3f7684a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.836+0000 7f3f7684a640 1 -- 172.21.15.22:0/2678949246 >> v1:172.21.15.33:6789/0 conn(0x7f3f7013f710 legacy=0x7f3f701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.836+0000 7f3f7684a640 1 -- 172.21.15.22:0/2678949246 shutdown_connections 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.836+0000 7f3f7684a640 1 -- 172.21.15.22:0/2678949246 >> 172.21.15.22:0/2678949246 conn(0x7f3f700b7ea0 msgr2=0x7f3f700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.836+0000 7f3f7684a640 1 -- 172.21.15.22:0/2678949246 shutdown_connections 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.836+0000 7f3f7684a640 1 -- 172.21.15.22:0/2678949246 wait complete. 2024-03-25T18:25:14.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 10 monclient: build_initial_monmap 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 10 monclient: monmap: 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 1 librados: starting msgr at 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 1 librados: starting objecter 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 1 Processor -- start 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 1 -- start start 2024-03-25T18:25:14.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 1 librados: setting wanted keys 2024-03-25T18:25:14.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 1 librados: calling monclient init 2024-03-25T18:25:14.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.837+0000 7f3f7684a640 10 monclient: init 2024-03-25T18:25:14.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:14.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:14.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 10 monclient(hunting): picked mon.noname-b con 0x7f3f700a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:14.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:14.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f701430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:14.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f70138a50 con 0x7f3f701430c0 2024-03-25T18:25:14.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f700ab9e0 con 0x7f3f700a98f0 2024-03-25T18:25:14.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f701d7880 con 0x7f3f7013f710 2024-03-25T18:25:14.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:14.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.838+0000 7f3f7684a640 10 monclient(hunting): authenticate will time out at 7514.061447s 2024-03-25T18:25:14.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.842+0000 7f3f6f7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f3f7013f710 0x7f3f701d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40948/0 (socket says 172.21.15.22:40948) 2024-03-25T18:25:14.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.842+0000 7f3f6f7fe640 1 -- 172.21.15.22:0/216267640 learned_addr learned my addr 172.21.15.22:0/216267640 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:14.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.843+0000 7f3f6cff9640 10 client.?.objecter ms_handle_connect 0x7f3f701430c0 2024-03-25T18:25:14.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.843+0000 7f3f6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:14.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.851+0000 7f3f6cff9640 10 client.?.objecter ms_handle_connect 0x7f3f7013f710 2024-03-25T18:25:14.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.851+0000 7f3f6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:14.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.857+0000 7f3f6cff9640 10 client.?.objecter ms_handle_connect 0x7f3f700a98f0 2024-03-25T18:25:14.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.857+0000 7f3f6cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:14.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.861+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881746105 0 0) 0x7f3f701d7880 con 0x7f3f7013f710 2024-03-25T18:25:14.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.861+0000 7f3f6cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.861+0000 7f3f6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.861+0000 7f3f6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.861+0000 7f3f6cff9640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:25:14.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.861+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f5c004eb0 con 0x7f3f7013f710 2024-03-25T18:25:14.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.862+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206674807 0 0) 0x7f3f700ab9e0 con 0x7f3f700a98f0 2024-03-25T18:25:14.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.862+0000 7f3f6cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.862+0000 7f3f6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.862+0000 7f3f6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.862+0000 7f3f6cff9640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:25:14.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.862+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f701d7880 con 0x7f3f700a98f0 2024-03-25T18:25:14.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.877+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3097038711 0 0) 0x7f3f5c004eb0 con 0x7f3f7013f710 2024-03-25T18:25:14.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.877+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f700ab9e0 con 0x7f3f7013f710 2024-03-25T18:25:14.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.880+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3f64001c40 con 0x7f3f7013f710 2024-03-25T18:25:14.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.880+0000 7f3f6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.880+0000 7f3f6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:14.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.880+0000 7f3f6cff9640 10 monclient(hunting): dump: 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:14.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.880+0000 7f3f6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:14.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.883+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50844299 0 0) 0x7f3f70138a50 con 0x7f3f701430c0 2024-03-25T18:25:14.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.883+0000 7f3f6cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.883+0000 7f3f6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.883+0000 7f3f6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.883+0000 7f3f6cff9640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:25:14.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.883+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f5c004eb0 con 0x7f3f701430c0 2024-03-25T18:25:14.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.887+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3307513760 0 0) 0x7f3f700ab9e0 con 0x7f3f7013f710 2024-03-25T18:25:14.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 >> v1:172.21.15.33:6789/0 conn(0x7f3f700a98f0 legacy=0x7f3f70138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 >> v1:172.21.15.22:6789/0 conn(0x7f3f701430c0 legacy=0x7f3f701e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:14.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 1 monclient: found mon.c 2024-03-25T18:25:14.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f701d84b0 con 0x7f3f7013f710 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 5 monclient: authenticate success, global_id 4496 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 10 monclient: _renew_subs 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f701d51f0 con 0x7f3f7013f710 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 10 monclient: _renew_subs 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f70138e60 con 0x7f3f7013f710 2024-03-25T18:25:14.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.888+0000 7f3f7684a640 1 librados: init done 2024-03-25T18:25:14.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.889+0000 7f3f7684a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:14.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.889+0000 7f3f7684a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:14.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.889+0000 7f3f7684a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:14.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:14.889+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f701d76e0 con 0x7f3f7013f710 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.201+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f64004a90 con 0x7f3f7013f710 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.201+0000 7f3f6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.201+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3f64004c00 con 0x7f3f7013f710 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.201+0000 7f3f6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.201+0000 7f3f6cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.201+0000 7f3f6cff9640 10 monclient: dump: 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:15.202 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.201+0000 7f3f6e7fc640 4 set_mon_vals no callback set 2024-03-25T18:25:15.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.202+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f3f6401abd0 con 0x7f3f7013f710 2024-03-25T18:25:15.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.203+0000 7f3f6ffff640 10 monclient: get_auth_request con 0x7f3f5c06ddb0 auth_method 0 2024-03-25T18:25:15.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.203+0000 7f3f6cff9640 10 client.4496.objecter ms_handle_connect 0x7f3f5c06ddb0 2024-03-25T18:25:15.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f3f640547c0 con 0x7f3f7013f710 2024-03-25T18:25:15.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 10 client.4496.objecter ms_dispatch 0x7f3f70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:15.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:15.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:15.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T18:25:15.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f64050100 con 0x7f3f7013f710 2024-03-25T18:25:15.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:15.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.209+0000 7f3f6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:15.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.448+0000 7f3f7684a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:15.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.448+0000 7f3f7684a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:15.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.448+0000 7f3f7684a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:15.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.448+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f701d7a10 con 0x7f3f7013f710 2024-03-25T18:25:15.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.477+0000 7f3f6cff9640 1 -- 172.21.15.22:0/216267640 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f3f64052200 con 0x7f3f7013f710 2024-03-25T18:25:15.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.477+0000 7f3f6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:15.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.477+0000 7f3f6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:15.478 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:15.478 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:15.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.479+0000 7f3f7684a640 10 librados: watch_flush enter 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 10 librados: watch_flush exit 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 10 client.4496.objecter successfully canceled tick 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 >> v1:172.21.15.33:6800/215350958 conn(0x7f3f5c06ddb0 legacy=0x7f3f5c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 10 monclient: shutdown 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 >> v1:172.21.15.22:6790/0 conn(0x7f3f7013f710 legacy=0x7f3f701d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 shutdown_connections 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 >> 172.21.15.22:0/216267640 conn(0x7f3f700b7ea0 msgr2=0x7f3f70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 shutdown_connections 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 1 -- 172.21.15.22:0/216267640 wait complete. 2024-03-25T18:25:15.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:15.480+0000 7f3f7684a640 1 librados: shutdown 2024-03-25T18:25:15.491 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:18.323 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:20.985 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.328 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:20.990 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.379 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:21.041 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.386 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:21.004 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.390 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:21.008 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.393 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:21.055 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.402 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:21.064 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.428 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:21.046 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.431 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:21.093 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:22.298 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:22.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.417+0000 7f5e94888640 10 monclient: get_monmap_and_config 2024-03-25T18:25:22.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.417+0000 7f5e94888640 10 monclient: build_initial_monmap 2024-03-25T18:25:22.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.417+0000 7f5e94888640 10 monclient: monmap: 2024-03-25T18:25:22.418 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:22.418 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:22.418 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:25:22.418201+0000 2024-03-25T18:25:22.418 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:25:22.418201+0000 2024-03-25T18:25:22.419 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:22.419 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:22.419 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:22.419 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:25:22.419 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:25:22.419 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:22.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.418+0000 7f5e94888640 1 Processor -- start 2024-03-25T18:25:22.420 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.418+0000 7f5e94888640 1 -- start start 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.418+0000 7f5e94888640 10 monclient: init 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.419+0000 7f5e94888640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.419+0000 7f5e94888640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.419+0000 7f5e94888640 10 monclient(hunting): picked mon.noname-a con 0x7f5e900a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.419+0000 7f5e94888640 10 monclient(hunting): picked mon.noname-b con 0x7f5e9013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.419+0000 7f5e94888640 10 monclient(hunting): picked mon.noname-c con 0x7f5e901430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.419+0000 7f5e94888640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e90142520 con 0x7f5e900a98f0 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.419+0000 7f5e94888640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900b28a0 con 0x7f5e901430c0 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.420+0000 7f5e94888640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900b2a30 con 0x7f5e9013f710 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.420+0000 7f5e94888640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:22.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.420+0000 7f5e94888640 10 monclient(hunting): authenticate will time out at 7521.642688s 2024-03-25T18:25:22.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.436+0000 7f5e8ed9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f5e901430c0 0x7f5e9014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:59684/0 (socket says 172.21.15.22:59684) 2024-03-25T18:25:22.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.437+0000 7f5e8ed9d640 1 -- 172.21.15.22:0/1702193464 learned_addr learned my addr 172.21.15.22:0/1702193464 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:22.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.456+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451196334 0 0) 0x7f5e900b28a0 con 0x7f5e901430c0 2024-03-25T18:25:22.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.456+0000 7f5e8d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.456+0000 7f5e8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.456+0000 7f5e8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.456+0000 7f5e8d59a640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:25:22.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.456+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e78003520 con 0x7f5e901430c0 2024-03-25T18:25:22.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465619071 0 0) 0x7f5e90142520 con 0x7f5e900a98f0 2024-03-25T18:25:22.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:25:22.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e900b28a0 con 0x7f5e900a98f0 2024-03-25T18:25:22.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080351337 0 0) 0x7f5e900b2a30 con 0x7f5e9013f710 2024-03-25T18:25:22.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:25:22.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.458+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e90142520 con 0x7f5e9013f710 2024-03-25T18:25:22.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.466+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991838021 0 0) 0x7f5e78003520 con 0x7f5e901430c0 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.466+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 >> v1:172.21.15.22:6790/0 conn(0x7f5e9013f710 legacy=0x7f5e901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 >> v1:172.21.15.22:6789/0 conn(0x7f5e900a98f0 legacy=0x7f5e900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 1 monclient: found mon.noname-c 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900b44c0 con 0x7f5e901430c0 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 10 monclient: _finish_auth 0 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5e88001a10 con 0x7f5e901430c0 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e8d59a640 10 monclient: dump: 2024-03-25T18:25:22.468 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e94888640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:25:22.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.467+0000 7f5e94888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:22.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e88001b70 con 0x7f5e901430c0 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e8d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e8d59a640 1 -- 172.21.15.22:0/1702193464 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5e88001f80 con 0x7f5e901430c0 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e8d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e8d59a640 10 monclient: dump: 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:22.479 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:22.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e94888640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:22.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.478+0000 7f5e94888640 4 set_mon_vals no callback set 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.479+0000 7f5e94888640 10 monclient: shutdown 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.479+0000 7f5e94888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.479+0000 7f5e94888640 1 -- 172.21.15.22:0/1702193464 >> v1:172.21.15.33:6789/0 conn(0x7f5e901430c0 legacy=0x7f5e9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.479+0000 7f5e94888640 1 -- 172.21.15.22:0/1702193464 shutdown_connections 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.479+0000 7f5e94888640 1 -- 172.21.15.22:0/1702193464 >> 172.21.15.22:0/1702193464 conn(0x7f5e900b7ea0 msgr2=0x7f5e900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.479+0000 7f5e94888640 1 -- 172.21.15.22:0/1702193464 shutdown_connections 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.479+0000 7f5e94888640 1 -- 172.21.15.22:0/1702193464 wait complete. 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.480+0000 7f5e94888640 10 monclient: build_initial_monmap 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.480+0000 7f5e94888640 10 monclient: monmap: 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:25:22.481 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 1 librados: starting msgr at 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 1 librados: starting objecter 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 1 Processor -- start 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 1 -- start start 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 1 librados: setting wanted keys 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 1 librados: calling monclient init 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 10 monclient: init 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 10 monclient(hunting): picked mon.noname-b con 0x7f5e900a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.481+0000 7f5e94888640 10 monclient(hunting): picked mon.noname-a con 0x7f5e9013f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.482+0000 7f5e94888640 10 monclient(hunting): picked mon.noname-c con 0x7f5e901430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.482+0000 7f5e94888640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e901369d0 con 0x7f5e9013f710 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.482+0000 7f5e94888640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e90133a60 con 0x7f5e900a98f0 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.482+0000 7f5e94888640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e900ab9e0 con 0x7f5e901430c0 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.482+0000 7f5e94888640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.482+0000 7f5e94888640 10 monclient(hunting): authenticate will time out at 7521.704665s 2024-03-25T18:25:22.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.489+0000 7f5e8e59c640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f5e900a98f0 0x7f5e90132c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:59742/0 (socket says 172.21.15.22:59742) 2024-03-25T18:25:22.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.489+0000 7f5e8e59c640 1 -- 172.21.15.22:0/2631795297 learned_addr learned my addr 172.21.15.22:0/2631795297 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:22.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.491+0000 7f5e7effd640 10 client.?.objecter ms_handle_connect 0x7f5e900a98f0 2024-03-25T18:25:22.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.491+0000 7f5e7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.491+0000 7f5e7effd640 10 client.?.objecter ms_handle_connect 0x7f5e901430c0 2024-03-25T18:25:22.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.491+0000 7f5e7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.498+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006707347 0 0) 0x7f5e90133a60 con 0x7f5e900a98f0 2024-03-25T18:25:22.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.498+0000 7f5e7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.498+0000 7f5e7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.498+0000 7f5e7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.498+0000 7f5e7effd640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:25:22.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.498+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e70004eb0 con 0x7f5e900a98f0 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.506+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638125381 0 0) 0x7f5e70004eb0 con 0x7f5e900a98f0 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.506+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e90133a60 con 0x7f5e900a98f0 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.506+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5e80001a30 con 0x7f5e900a98f0 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.506+0000 7f5e7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.506+0000 7f5e7effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.506+0000 7f5e7effd640 10 monclient(hunting): dump: 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:22.507 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.506+0000 7f5e7effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.508+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019601797 0 0) 0x7f5e900ab9e0 con 0x7f5e901430c0 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.508+0000 7f5e7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.508+0000 7f5e7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.508+0000 7f5e7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.508+0000 7f5e7effd640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:25:22.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.508+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e70004eb0 con 0x7f5e901430c0 2024-03-25T18:25:22.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3860756850 0 0) 0x7f5e90133a60 con 0x7f5e900a98f0 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 >> v1:172.21.15.22:6790/0 conn(0x7f5e901430c0 legacy=0x7f5e90133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 >> v1:172.21.15.22:6789/0 conn(0x7f5e9013f710 legacy=0x7f5e901362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 1 monclient: found mon.b 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e900abcc0 con 0x7f5e900a98f0 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 10 monclient: _finish_auth 0 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 10 monclient: _renew_subs 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e900abfa0 con 0x7f5e900a98f0 2024-03-25T18:25:22.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 10 monclient: _renew_subs 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e90137e40 con 0x7f5e900a98f0 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.524+0000 7f5e94888640 1 librados: init done 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.525+0000 7f5e94888640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.525+0000 7f5e94888640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.525+0000 7f5e94888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:22.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.525+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e900627b0 con 0x7f5e900a98f0 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.804+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e80001d10 con 0x7f5e900a98f0 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.804+0000 7f5e7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.804+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5e80003a60 con 0x7f5e900a98f0 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.804+0000 7f5e7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.804+0000 7f5e8cd99640 4 set_mon_vals no callback set 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.804+0000 7f5e7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.804+0000 7f5e7effd640 10 monclient: dump: 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:22.805 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:22.806 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:22.806 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:22.806 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:22.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.805+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f5e80003c80 con 0x7f5e900a98f0 2024-03-25T18:25:22.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.805+0000 7f5e8dd9b640 10 monclient: get_auth_request con 0x7f5e7006dd30 auth_method 0 2024-03-25T18:25:22.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.805+0000 7f5e7effd640 10 client.4498.objecter ms_handle_connect 0x7f5e7006dd30 2024-03-25T18:25:22.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.811+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f5e800548e0 con 0x7f5e900a98f0 2024-03-25T18:25:22.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.811+0000 7f5e7effd640 10 client.4498.objecter ms_dispatch 0x7f5e90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:22.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.811+0000 7f5e7effd640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:22.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.811+0000 7f5e7effd640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:22.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.811+0000 7f5e7effd640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T18:25:22.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.813+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e800503f0 con 0x7f5e900a98f0 2024-03-25T18:25:22.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.813+0000 7f5e7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:22.813+0000 7f5e7effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:23.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.051+0000 7f5e94888640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:23.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.051+0000 7f5e94888640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:23.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.051+0000 7f5e94888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:23.052 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.051+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e9013dfd0 con 0x7f5e900a98f0 2024-03-25T18:25:23.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.131+0000 7f5e7effd640 1 -- 172.21.15.22:0/2631795297 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f5e80052320 con 0x7f5e900a98f0 2024-03-25T18:25:23.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.131+0000 7f5e7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:23.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.131+0000 7f5e7effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:23.131 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:23.132 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:23.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 10 librados: watch_flush enter 2024-03-25T18:25:23.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 10 librados: watch_flush exit 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 10 client.4498.objecter successfully canceled tick 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 >> v1:172.21.15.33:6800/215350958 conn(0x7f5e7006dd30 legacy=0x7f5e700701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 10 monclient: shutdown 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 >> v1:172.21.15.33:6789/0 conn(0x7f5e900a98f0 legacy=0x7f5e90132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 shutdown_connections 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 >> 172.21.15.22:0/2631795297 conn(0x7f5e900b7ea0 msgr2=0x7f5e90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 shutdown_connections 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 1 -- 172.21.15.22:0/2631795297 wait complete. 2024-03-25T18:25:23.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:23.133+0000 7f5e94888640 1 librados: shutdown 2024-03-25T18:25:23.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:29.952 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:30.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.070+0000 7faf3767b640 10 monclient: get_monmap_and_config 2024-03-25T18:25:30.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.070+0000 7faf3767b640 10 monclient: build_initial_monmap 2024-03-25T18:25:30.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.070+0000 7faf3767b640 10 monclient: monmap: 2024-03-25T18:25:30.073 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:30.073 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:30.073 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:25:30.072157+0000 2024-03-25T18:25:30.073 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:25:30.072157+0000 2024-03-25T18:25:30.073 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:30.073 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:30.074 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:30.074 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:25:30.074 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:25:30.074 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:30.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.071+0000 7faf3767b640 1 Processor -- start 2024-03-25T18:25:30.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.071+0000 7faf3767b640 1 -- start start 2024-03-25T18:25:30.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.071+0000 7faf3767b640 10 monclient: init 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 10 monclient(hunting): picked mon.noname-c con 0x7faf300a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 10 monclient(hunting): picked mon.noname-b con 0x7faf3013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 10 monclient(hunting): picked mon.noname-a con 0x7faf301430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30142520 con 0x7faf301430c0 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300b28a0 con 0x7faf300a98f0 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf300b2a30 con 0x7faf3013f710 2024-03-25T18:25:30.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:30.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.072+0000 7faf3767b640 10 monclient(hunting): authenticate will time out at 7529.296174s 2024-03-25T18:25:30.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.086+0000 7faf34c16640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7faf3013f710 0x7faf301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47398/0 (socket says 172.21.15.22:47398) 2024-03-25T18:25:30.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.086+0000 7faf34c16640 1 -- 172.21.15.22:0/3037611998 learned_addr learned my addr 172.21.15.22:0/3037611998 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:30.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.104+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27760901 0 0) 0x7faf30142520 con 0x7faf301430c0 2024-03-25T18:25:30.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.104+0000 7faf27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:30.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.104+0000 7faf27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:30.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.104+0000 7faf27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:30.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.104+0000 7faf27fff640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:25:30.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.104+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf1c003520 con 0x7faf301430c0 2024-03-25T18:25:30.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.105+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260264282 0 0) 0x7faf300b28a0 con 0x7faf300a98f0 2024-03-25T18:25:30.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.105+0000 7faf27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:30.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.105+0000 7faf27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:30.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.105+0000 7faf27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:30.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.105+0000 7faf27fff640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:25:30.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.105+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf30142520 con 0x7faf300a98f0 2024-03-25T18:25:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298304826 0 0) 0x7faf1c003520 con 0x7faf301430c0 2024-03-25T18:25:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 >> v1:172.21.15.22:6790/0 conn(0x7faf3013f710 legacy=0x7faf301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 >> v1:172.21.15.33:6789/0 conn(0x7faf300a98f0 legacy=0x7faf300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 1 monclient: found mon.noname-a 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf300b44c0 con 0x7faf301430c0 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faf28001a10 con 0x7faf301430c0 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf3767b640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf3767b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf27fff640 10 monclient: dump: 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:30.116 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:30.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.114+0000 7faf3767b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf28001b70 con 0x7faf301430c0 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf27fff640 1 -- 172.21.15.22:0/3037611998 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faf28001f80 con 0x7faf301430c0 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf3767b640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf3767b640 4 set_mon_vals no callback set 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf27fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.131+0000 7faf27fff640 10 monclient: dump: 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:30.133 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:30.134 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:30.134 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:30.134 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:30.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.132+0000 7faf3767b640 10 monclient: shutdown 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.132+0000 7faf3767b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.132+0000 7faf3767b640 1 -- 172.21.15.22:0/3037611998 >> v1:172.21.15.22:6789/0 conn(0x7faf301430c0 legacy=0x7faf3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.132+0000 7faf3767b640 1 -- 172.21.15.22:0/3037611998 shutdown_connections 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.132+0000 7faf3767b640 1 -- 172.21.15.22:0/3037611998 >> 172.21.15.22:0/3037611998 conn(0x7faf300b7ea0 msgr2=0x7faf300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.132+0000 7faf3767b640 1 -- 172.21.15.22:0/3037611998 shutdown_connections 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.132+0000 7faf3767b640 1 -- 172.21.15.22:0/3037611998 wait complete. 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 10 monclient: build_initial_monmap 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 10 monclient: monmap: 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:30.136 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 1 librados: starting msgr at 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 1 librados: starting objecter 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 1 Processor -- start 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 1 -- start start 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 1 librados: setting wanted keys 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 1 librados: calling monclient init 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.133+0000 7faf3767b640 10 monclient: init 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 10 monclient(hunting): picked mon.noname-a con 0x7faf300a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:30.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 10 monclient(hunting): picked mon.noname-c con 0x7faf3013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 10 monclient(hunting): picked mon.noname-b con 0x7faf301430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf30139830 con 0x7faf300a98f0 2024-03-25T18:25:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf301db7d0 con 0x7faf301430c0 2024-03-25T18:25:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf301db410 con 0x7faf3013f710 2024-03-25T18:25:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:30.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.134+0000 7faf3767b640 10 monclient(hunting): authenticate will time out at 7529.358340s 2024-03-25T18:25:30.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.143+0000 7faf35c18640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7faf301430c0 0x7faf301d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:60358/0 (socket says 172.21.15.22:60358) 2024-03-25T18:25:30.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.143+0000 7faf35c18640 1 -- 172.21.15.22:0/398299514 learned_addr learned my addr 172.21.15.22:0/398299514 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:30.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.148+0000 7faf25ffb640 10 client.?.objecter ms_handle_connect 0x7faf301430c0 2024-03-25T18:25:30.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.148+0000 7faf25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:30.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.160+0000 7faf25ffb640 10 client.?.objecter ms_handle_connect 0x7faf3013f710 2024-03-25T18:25:30.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.160+0000 7faf25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:30.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.166+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 352016156 0 0) 0x7faf301db7d0 con 0x7faf301430c0 2024-03-25T18:25:30.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.166+0000 7faf25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:30.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.166+0000 7faf25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:30.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.166+0000 7faf25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:30.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.166+0000 7faf25ffb640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:25:30.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.166+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf14004eb0 con 0x7faf301430c0 2024-03-25T18:25:30.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.169+0000 7faf25ffb640 10 client.?.objecter ms_handle_connect 0x7faf300a98f0 2024-03-25T18:25:30.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.169+0000 7faf25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:30.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.177+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702008001 0 0) 0x7faf301db410 con 0x7faf3013f710 2024-03-25T18:25:30.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.177+0000 7faf25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:30.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.177+0000 7faf25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:30.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.177+0000 7faf25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:30.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.177+0000 7faf25ffb640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:25:30.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.177+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf301db7d0 con 0x7faf3013f710 2024-03-25T18:25:30.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.186+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345557345 0 0) 0x7faf30139830 con 0x7faf300a98f0 2024-03-25T18:25:30.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.186+0000 7faf25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:30.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.186+0000 7faf25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:30.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.186+0000 7faf25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:30.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.186+0000 7faf25ffb640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:25:30.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.186+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf301db410 con 0x7faf300a98f0 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.190+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4220980657 0 0) 0x7faf14004eb0 con 0x7faf301430c0 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.191+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf30139830 con 0x7faf301430c0 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.191+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faf28001f80 con 0x7faf301430c0 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.191+0000 7faf25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.191+0000 7faf25ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.191+0000 7faf25ffb640 10 monclient(hunting): dump: 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:30.192 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:30.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.191+0000 7faf25ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.193+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1511864529 0 0) 0x7faf301db7d0 con 0x7faf3013f710 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.193+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf14004eb0 con 0x7faf3013f710 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.193+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faf200018a0 con 0x7faf3013f710 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.193+0000 7faf25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.193+0000 7faf25ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.193+0000 7faf25ffb640 10 monclient(hunting): dump: 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:30.195 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:30.196 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:30.196 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:30.196 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:30.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 253960732 0 0) 0x7faf30139830 con 0x7faf301430c0 2024-03-25T18:25:30.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 >> v1:172.21.15.22:6790/0 conn(0x7faf3013f710 legacy=0x7faf300b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 >> v1:172.21.15.22:6789/0 conn(0x7faf300a98f0 legacy=0x7faf301dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:30.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 1 monclient: found mon.b 2024-03-25T18:25:30.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf301db5a0 con 0x7faf301430c0 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 10 monclient: _renew_subs 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf300b1480 con 0x7faf301430c0 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 10 monclient: _renew_subs 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.212+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf300b0b10 con 0x7faf301430c0 2024-03-25T18:25:30.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.213+0000 7faf3767b640 1 librados: init done 2024-03-25T18:25:30.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.213+0000 7faf3767b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:30.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.213+0000 7faf3767b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:30.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.213+0000 7faf3767b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:30.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.213+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf300627b0 con 0x7faf301430c0 2024-03-25T18:25:30.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.504+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf28003c20 con 0x7faf301430c0 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.504+0000 7faf25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.504+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faf28003ff0 con 0x7faf301430c0 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.504+0000 7faf25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.504+0000 7faf277fe640 4 set_mon_vals no callback set 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.504+0000 7faf25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.504+0000 7faf25ffb640 10 monclient: dump: 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:30.506 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:30.507 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:30.507 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:30.507 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:30.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.505+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7faf28004210 con 0x7faf301430c0 2024-03-25T18:25:30.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.505+0000 7faf35417640 10 monclient: get_auth_request con 0x7faf1406df70 auth_method 0 2024-03-25T18:25:30.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.505+0000 7faf25ffb640 10 client.4504.objecter ms_handle_connect 0x7faf1406df70 2024-03-25T18:25:30.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.510+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7faf28054bc0 con 0x7faf301430c0 2024-03-25T18:25:30.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.510+0000 7faf25ffb640 10 client.4504.objecter ms_dispatch 0x7faf30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:30.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.510+0000 7faf25ffb640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:30.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.510+0000 7faf25ffb640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:30.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.510+0000 7faf25ffb640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T18:25:30.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.512+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf280506d0 con 0x7faf301430c0 2024-03-25T18:25:30.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.512+0000 7faf25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:30.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.512+0000 7faf25ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.732+0000 7faf3767b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.732+0000 7faf3767b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.732+0000 7faf3767b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.732+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf30069f40 con 0x7faf301430c0 2024-03-25T18:25:30.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.763+0000 7faf25ffb640 1 -- 172.21.15.22:0/398299514 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7faf28052600 con 0x7faf301430c0 2024-03-25T18:25:30.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.763+0000 7faf25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:30.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.763+0000 7faf25ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:30.765 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:30.765 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:30.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 10 librados: watch_flush enter 2024-03-25T18:25:30.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 10 librados: watch_flush exit 2024-03-25T18:25:30.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T18:25:30.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 10 client.4504.objecter successfully canceled tick 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 >> v1:172.21.15.33:6800/215350958 conn(0x7faf1406df70 legacy=0x7faf14070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 10 monclient: shutdown 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.765+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 >> v1:172.21.15.33:6789/0 conn(0x7faf301430c0 legacy=0x7faf301d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.766+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 shutdown_connections 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.766+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 >> 172.21.15.22:0/398299514 conn(0x7faf300b7ea0 msgr2=0x7faf30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.766+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 shutdown_connections 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.766+0000 7faf3767b640 1 -- 172.21.15.22:0/398299514 wait complete. 2024-03-25T18:25:30.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:30.766+0000 7faf3767b640 1 librados: shutdown 2024-03-25T18:25:30.778 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:37.585 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:37.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.701+0000 7f6d7afd6640 10 monclient: get_monmap_and_config 2024-03-25T18:25:37.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.701+0000 7f6d7afd6640 10 monclient: build_initial_monmap 2024-03-25T18:25:37.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.701+0000 7f6d7afd6640 10 monclient: monmap: 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:25:37.702588+0000 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:25:37.702588+0000 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:25:37.703 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:37.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 1 Processor -- start 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 1 -- start start 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient: init 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient(hunting): picked mon.noname-b con 0x7f6d7406ab80 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient(hunting): picked mon.noname-c con 0x7f6d7406af60 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d740b00b0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740b0500 con 0x7f6d740b00b0 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7413ffa0 con 0x7f6d7406af60 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740af520 con 0x7f6d7406ab80 2024-03-25T18:25:37.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:37.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.702+0000 7f6d7afd6640 10 monclient(hunting): authenticate will time out at 7536.926407s 2024-03-25T18:25:37.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.716+0000 7f6d73fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f6d7406af60 0x7f6d7413f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:49184/0 (socket says 172.21.15.22:49184) 2024-03-25T18:25:37.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.716+0000 7f6d73fff640 1 -- 172.21.15.22:0/3300105739 learned_addr learned my addr 172.21.15.22:0/3300105739 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:37.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.728+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092666351 0 0) 0x7f6d7413ffa0 con 0x7f6d7406af60 2024-03-25T18:25:37.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.728+0000 7f6d737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.728+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.728+0000 7f6d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.728+0000 7f6d737fe640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:25:37.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.728+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d64003520 con 0x7f6d7406af60 2024-03-25T18:25:37.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.733+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712837985 0 0) 0x7f6d740af520 con 0x7f6d7406ab80 2024-03-25T18:25:37.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.733+0000 7f6d737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.733+0000 7f6d737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.733+0000 7f6d737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.733+0000 7f6d737fe640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:25:37.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.733+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7413ffa0 con 0x7f6d7406ab80 2024-03-25T18:25:37.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1056489738 0 0) 0x7f6d64003520 con 0x7f6d7406af60 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 >> v1:172.21.15.22:6790/0 conn(0x7f6d7406ab80 legacy=0x7f6d740ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 >> v1:172.21.15.22:6789/0 conn(0x7f6d740b00b0 legacy=0x7f6d7414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d74072890 con 0x7f6d7406af60 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6d60001690 con 0x7f6d7406af60 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d737fe640 10 monclient: dump: 2024-03-25T18:25:37.743 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d7afd6640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:25:37.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.741+0000 7f6d7afd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d600017f0 con 0x7f6d7406af60 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d737fe640 1 -- 172.21.15.22:0/3300105739 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6d60001c00 con 0x7f6d7406af60 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 4 set_mon_vals no callback set 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:37.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d737fe640 10 monclient: dump: 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:37.748 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:37.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 10 monclient: shutdown 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 1 -- 172.21.15.22:0/3300105739 >> v1:172.21.15.33:6789/0 conn(0x7f6d7406af60 legacy=0x7f6d7413f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 1 -- 172.21.15.22:0/3300105739 shutdown_connections 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 1 -- 172.21.15.22:0/3300105739 >> 172.21.15.22:0/3300105739 conn(0x7f6d740b4a80 msgr2=0x7f6d740b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.746+0000 7f6d7afd6640 1 -- 172.21.15.22:0/3300105739 shutdown_connections 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.747+0000 7f6d7afd6640 1 -- 172.21.15.22:0/3300105739 wait complete. 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.747+0000 7f6d7afd6640 10 monclient: build_initial_monmap 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.747+0000 7f6d7afd6640 10 monclient: monmap: 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:37.751 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.747+0000 7f6d7afd6640 1 librados: starting msgr at 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.747+0000 7f6d7afd6640 1 librados: starting objecter 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 1 Processor -- start 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 1 -- start start 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 1 librados: setting wanted keys 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 1 librados: calling monclient init 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 10 monclient: init 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d7406ab80 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 10 monclient(hunting): picked mon.noname-b con 0x7f6d740b00b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 10 monclient(hunting): picked mon.noname-c con 0x7f6d741d80d0 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d741453b0 con 0x7f6d7406ab80 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.748+0000 7f6d7afd6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d74145600 con 0x7f6d740b00b0 2024-03-25T18:25:37.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.749+0000 7f6d7afd6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d74143820 con 0x7f6d741d80d0 2024-03-25T18:25:37.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.749+0000 7f6d7afd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:37.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.749+0000 7f6d7afd6640 10 monclient(hunting): authenticate will time out at 7536.972601s 2024-03-25T18:25:37.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.767+0000 7f6d78d72640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6d7406ab80 0x7f6d741d79c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:57030/0 (socket says 172.21.15.22:57030) 2024-03-25T18:25:37.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.767+0000 7f6d78d72640 1 -- 172.21.15.22:0/225674928 learned_addr learned my addr 172.21.15.22:0/225674928 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:37.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.769+0000 7f6d717fa640 10 client.?.objecter ms_handle_connect 0x7f6d7406ab80 2024-03-25T18:25:37.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.769+0000 7f6d717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:37.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.769+0000 7f6d717fa640 10 client.?.objecter ms_handle_connect 0x7f6d740b00b0 2024-03-25T18:25:37.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.769+0000 7f6d717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:37.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.776+0000 7f6d717fa640 10 client.?.objecter ms_handle_connect 0x7f6d741d80d0 2024-03-25T18:25:37.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.776+0000 7f6d717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:37.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.782+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992300768 0 0) 0x7f6d741453b0 con 0x7f6d7406ab80 2024-03-25T18:25:37.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.782+0000 7f6d717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.782+0000 7f6d717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.782+0000 7f6d717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.782+0000 7f6d717fa640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:25:37.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.782+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d58004eb0 con 0x7f6d7406ab80 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863473210 0 0) 0x7f6d74143820 con 0x7f6d741d80d0 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d741453b0 con 0x7f6d741d80d0 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528819805 0 0) 0x7f6d74145600 con 0x7f6d740b00b0 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:25:37.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.786+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d74143820 con 0x7f6d740b00b0 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.804+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 871268691 0 0) 0x7f6d58004eb0 con 0x7f6d7406ab80 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.804+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d74145600 con 0x7f6d7406ab80 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.804+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6d68001a10 con 0x7f6d7406ab80 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.804+0000 7f6d717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.804+0000 7f6d717fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.804+0000 7f6d717fa640 10 monclient(hunting): dump: 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:37.806 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:37.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.804+0000 7f6d717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.820+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2219141871 0 0) 0x7f6d74143820 con 0x7f6d740b00b0 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.820+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d58004eb0 con 0x7f6d740b00b0 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.820+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6d60001b00 con 0x7f6d740b00b0 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.820+0000 7f6d717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.820+0000 7f6d717fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.820+0000 7f6d717fa640 10 monclient(hunting): dump: 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.822 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:37.823 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:37.823 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:37.823 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:37.823 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.832+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2376321323 0 0) 0x7f6d741453b0 con 0x7f6d741d80d0 2024-03-25T18:25:37.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.832+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d74143820 con 0x7f6d741d80d0 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.832+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6d6c001b10 con 0x7f6d741d80d0 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.832+0000 7f6d717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.832+0000 7f6d717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.832+0000 7f6d717fa640 10 monclient(hunting): dump: 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:37.834 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:37.835 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:37.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1913664509 0 0) 0x7f6d74145600 con 0x7f6d7406ab80 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 >> v1:172.21.15.22:6790/0 conn(0x7f6d741d80d0 legacy=0x7f6d741e2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 >> v1:172.21.15.33:6789/0 conn(0x7f6d740b00b0 legacy=0x7f6d74145de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 1 monclient: found mon.a 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d741d74d0 con 0x7f6d7406ab80 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 10 monclient: _renew_subs 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d741d5010 con 0x7f6d7406ab80 2024-03-25T18:25:37.837 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 10 monclient: _renew_subs 2024-03-25T18:25:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d74143460 con 0x7f6d7406ab80 2024-03-25T18:25:37.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.835+0000 7f6d7afd6640 1 librados: init done 2024-03-25T18:25:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.836+0000 7f6d7afd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.836+0000 7f6d7afd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.836+0000 7f6d7afd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:37.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:37.836+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7413ba20 con 0x7f6d7406ab80 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.170+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d68001c50 con 0x7f6d7406ab80 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.170+0000 7f6d717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.170+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6d68003a60 con 0x7f6d7406ab80 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.170+0000 7f6d717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.170+0000 7f6d717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.170+0000 7f6d717fa640 10 monclient: dump: 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:38.172 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:38.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.170+0000 7f6d72ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:38.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.171+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f6d68003c80 con 0x7f6d7406ab80 2024-03-25T18:25:38.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.172+0000 7f6d73fff640 10 monclient: get_auth_request con 0x7f6d5806e0f0 auth_method 0 2024-03-25T18:25:38.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.172+0000 7f6d717fa640 10 client.4503.objecter ms_handle_connect 0x7f6d5806e0f0 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f6d680548b0 con 0x7f6d7406ab80 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 10 client.4503.objecter ms_dispatch 0x7f6d74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d680503c0 con 0x7f6d7406ab80 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:38.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.178+0000 7f6d717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:38.380 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:41.042 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.397+0000 7f6d7afd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:38.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.397+0000 7f6d7afd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:38.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.397+0000 7f6d7afd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:25:38.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.397+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d740627b0 con 0x7f6d7406ab80 2024-03-25T18:25:38.455 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:41.117 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.454+0000 7f6d717fa640 1 -- 172.21.15.22:0/225674928 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f6d680522f0 con 0x7f6d7406ab80 2024-03-25T18:25:38.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.454+0000 7f6d717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:38.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.454+0000 7f6d717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:38.456 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:38.456 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:38.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 10 librados: watch_flush enter 2024-03-25T18:25:38.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 10 librados: watch_flush exit 2024-03-25T18:25:38.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-25T18:25:38.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 10 client.4503.objecter successfully canceled tick 2024-03-25T18:25:38.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 >> v1:172.21.15.33:6800/215350958 conn(0x7f6d5806e0f0 legacy=0x7f6d580705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:38.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 10 monclient: shutdown 2024-03-25T18:25:38.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:38.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 >> v1:172.21.15.22:6789/0 conn(0x7f6d7406ab80 legacy=0x7f6d741d79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:38.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 shutdown_connections 2024-03-25T18:25:38.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 >> 172.21.15.22:0/225674928 conn(0x7f6d740b4a80 msgr2=0x7f6d740ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:38.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 shutdown_connections 2024-03-25T18:25:38.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 1 -- 172.21.15.22:0/225674928 wait complete. 2024-03-25T18:25:38.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:38.456+0000 7f6d7afd6640 1 librados: shutdown 2024-03-25T18:25:38.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:38.486 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:06:41.148 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.491 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:41.109 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.493 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:41.155 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.495 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:41.113 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.503 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:41.165 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.521 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:06:41.139 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:38.534 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:06:41.196 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:45.275 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.389+0000 7f30af56c640 10 monclient: get_monmap_and_config 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.389+0000 7f30af56c640 10 monclient: build_initial_monmap 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.389+0000 7f30af56c640 10 monclient: monmap: 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:25:45.390775+0000 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:25:45.390775+0000 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:25:45.391 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.390+0000 7f30af56c640 1 Processor -- start 2024-03-25T18:25:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.390+0000 7f30af56c640 1 -- start start 2024-03-25T18:25:45.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.390+0000 7f30af56c640 10 monclient: init 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 10 monclient(hunting): picked mon.noname-a con 0x7f30a80a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 10 monclient(hunting): picked mon.noname-c con 0x7f30a813f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 10 monclient(hunting): picked mon.noname-b con 0x7f30a81430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a8142520 con 0x7f30a80a98f0 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80b28a0 con 0x7f30a813f710 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80b2a30 con 0x7f30a81430c0 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:45.393 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.391+0000 7f30af56c640 10 monclient(hunting): authenticate will time out at 7544.614847s 2024-03-25T18:25:45.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.400+0000 7f30acb07640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f30a813f710 0x7f30a81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:57924/0 (socket says 172.21.15.22:57924) 2024-03-25T18:25:45.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.400+0000 7f30acb07640 1 -- 172.21.15.22:0/594068679 learned_addr learned my addr 172.21.15.22:0/594068679 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:45.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.418+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103722823 0 0) 0x7f30a80b28a0 con 0x7f30a813f710 2024-03-25T18:25:45.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.418+0000 7f309ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:45.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.418+0000 7f309ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:45.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.418+0000 7f309ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:45.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.418+0000 7f309ffff640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:25:45.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.418+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3094003520 con 0x7f30a813f710 2024-03-25T18:25:45.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.428+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270689594 0 0) 0x7f30a80b2a30 con 0x7f30a81430c0 2024-03-25T18:25:45.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:45.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:45.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:45.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:25:45.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a80b28a0 con 0x7f30a81430c0 2024-03-25T18:25:45.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714024547 0 0) 0x7f30a8142520 con 0x7f30a80a98f0 2024-03-25T18:25:45.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:45.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:45.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:45.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:25:45.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.429+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a80b2a30 con 0x7f30a80a98f0 2024-03-25T18:25:45.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.437+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3789655623 0 0) 0x7f3094003520 con 0x7f30a813f710 2024-03-25T18:25:45.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.437+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 >> v1:172.21.15.22:6790/0 conn(0x7f30a81430c0 legacy=0x7f30a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.437+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 >> v1:172.21.15.22:6789/0 conn(0x7f30a80a98f0 legacy=0x7f30a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.437+0000 7f309ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.437+0000 7f309ffff640 1 monclient: found mon.noname-c 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.437+0000 7f309ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a80b44c0 con 0x7f30a813f710 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f30af56c640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f30af56c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3098001690 con 0x7f30a813f710 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f309ffff640 10 monclient: dump: 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.440 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.441 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:45.441 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:45.441 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:45.441 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:45.441 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:45.441 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:45.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.438+0000 7f30af56c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:45.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30980018b0 con 0x7f30a813f710 2024-03-25T18:25:45.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f309ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:45.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f309ffff640 1 -- 172.21.15.22:0/594068679 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3098001c40 con 0x7f30a813f710 2024-03-25T18:25:45.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f30af56c640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:45.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f30af56c640 4 set_mon_vals no callback set 2024-03-25T18:25:45.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f309ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f309ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.468+0000 7f309ffff640 10 monclient: dump: 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:45.470 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:45.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.469+0000 7f30af56c640 10 monclient: shutdown 2024-03-25T18:25:45.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.469+0000 7f30af56c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:45.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.469+0000 7f30af56c640 1 -- 172.21.15.22:0/594068679 >> v1:172.21.15.33:6789/0 conn(0x7f30a813f710 legacy=0x7f30a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.469+0000 7f30af56c640 1 -- 172.21.15.22:0/594068679 shutdown_connections 2024-03-25T18:25:45.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.469+0000 7f30af56c640 1 -- 172.21.15.22:0/594068679 >> 172.21.15.22:0/594068679 conn(0x7f30a80b7ea0 msgr2=0x7f30a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:45.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.469+0000 7f30af56c640 1 -- 172.21.15.22:0/594068679 shutdown_connections 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.469+0000 7f30af56c640 1 -- 172.21.15.22:0/594068679 wait complete. 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 10 monclient: build_initial_monmap 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 10 monclient: monmap: 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 1 librados: starting msgr at 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 1 librados: starting objecter 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 1 Processor -- start 2024-03-25T18:25:45.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 1 -- start start 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 1 librados: setting wanted keys 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 1 librados: calling monclient init 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.470+0000 7f30af56c640 10 monclient: init 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 10 monclient(hunting): picked mon.noname-b con 0x7f30a80a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 10 monclient(hunting): picked mon.noname-c con 0x7f30a813f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 10 monclient(hunting): picked mon.noname-a con 0x7f30a81430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a8138a50 con 0x7f30a81430c0 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a80ab9e0 con 0x7f30a80a98f0 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a81d7880 con 0x7f30a813f710 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:45.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.471+0000 7f30af56c640 10 monclient(hunting): authenticate will time out at 7544.694833s 2024-03-25T18:25:45.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.476+0000 7f30ad308640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f30a80a98f0 0x7f30a8138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:57978/0 (socket says 172.21.15.22:57978) 2024-03-25T18:25:45.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.476+0000 7f30ad308640 1 -- 172.21.15.22:0/2341964341 learned_addr learned my addr 172.21.15.22:0/2341964341 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:45.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.476+0000 7f309dffb640 10 client.?.objecter ms_handle_connect 0x7f30a80a98f0 2024-03-25T18:25:45.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.476+0000 7f309dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:45.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.477+0000 7f309dffb640 10 client.?.objecter ms_handle_connect 0x7f30a813f710 2024-03-25T18:25:45.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.477+0000 7f309dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:45.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.480+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136698127 0 0) 0x7f30a81d7880 con 0x7f30a813f710 2024-03-25T18:25:45.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.480+0000 7f309dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:45.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.480+0000 7f309dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:45.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.480+0000 7f309dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:45.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.480+0000 7f309dffb640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:25:45.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.480+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c004eb0 con 0x7f30a813f710 2024-03-25T18:25:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.487+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2934361660 0 0) 0x7f308c004eb0 con 0x7f30a813f710 2024-03-25T18:25:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.487+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30a81d7880 con 0x7f30a813f710 2024-03-25T18:25:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.487+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f30980015d0 con 0x7f30a813f710 2024-03-25T18:25:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.487+0000 7f309dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.487+0000 7f309dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.487+0000 7f309dffb640 10 monclient(hunting): dump: 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:45.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.487+0000 7f309dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:45.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.489+0000 7f309dffb640 10 client.?.objecter ms_handle_connect 0x7f30a81430c0 2024-03-25T18:25:45.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.489+0000 7f309dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.490+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244104879 0 0) 0x7f30a80ab9e0 con 0x7f30a80a98f0 2024-03-25T18:25:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.490+0000 7f309dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.490+0000 7f309dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.490+0000 7f309dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.490+0000 7f309dffb640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:25:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.490+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c004eb0 con 0x7f30a80a98f0 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1687953044 0 0) 0x7f30a81d7880 con 0x7f30a813f710 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 >> v1:172.21.15.33:6789/0 conn(0x7f30a80a98f0 legacy=0x7f30a8138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 >> v1:172.21.15.22:6789/0 conn(0x7f30a81430c0 legacy=0x7f30a81e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 1 monclient: found mon.c 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a81d84b0 con 0x7f30a813f710 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f309dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 5 monclient: authenticate success, global_id 4520 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 10 monclient: _renew_subs 2024-03-25T18:25:45.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a81d51f0 con 0x7f30a813f710 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 10 monclient: _renew_subs 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a8138e60 con 0x7f30a813f710 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.495+0000 7f30af56c640 1 librados: init done 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.496+0000 7f30af56c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.496+0000 7f30af56c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.496+0000 7f30af56c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.496+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a81d76e0 con 0x7f30a813f710 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.820+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3098004a90 con 0x7f30a813f710 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.820+0000 7f309dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.820+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3098004c00 con 0x7f30a813f710 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.820+0000 7f309dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.820+0000 7f309dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.820+0000 7f309dffb640 10 monclient: dump: 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:45.822 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:45.823 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:45.823 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:45.823 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:45.823 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:45.823 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:45.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.820+0000 7f309f7fe640 4 set_mon_vals no callback set 2024-03-25T18:25:45.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.821+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f309801abd0 con 0x7f30a813f710 2024-03-25T18:25:45.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.821+0000 7f30ad308640 10 monclient: get_auth_request con 0x7f308c06dd90 auth_method 0 2024-03-25T18:25:45.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.822+0000 7f309dffb640 10 client.4520.objecter ms_handle_connect 0x7f308c06dd90 2024-03-25T18:25:45.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f3098054700 con 0x7f30a813f710 2024-03-25T18:25:45.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 10 client.4520.objecter ms_dispatch 0x7f30a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:45.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:45.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:45.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T18:25:45.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3098050210 con 0x7f30a813f710 2024-03-25T18:25:45.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:45.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:45.827+0000 7f309dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:46.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.056+0000 7f30af56c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:46.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.056+0000 7f30af56c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:46.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.056+0000 7f30af56c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:25:46.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.056+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30a81d7a10 con 0x7f30a813f710 2024-03-25T18:25:46.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.080+0000 7f309dffb640 1 -- 172.21.15.22:0/2341964341 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f3098052140 con 0x7f30a813f710 2024-03-25T18:25:46.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.080+0000 7f309dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:46.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.080+0000 7f309dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:46.083 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:46.083 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 10 librados: watch_flush enter 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 10 librados: watch_flush exit 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 10 client.4520.objecter successfully canceled tick 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 >> v1:172.21.15.33:6800/215350958 conn(0x7f308c06dd90 legacy=0x7f308c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 10 monclient: shutdown 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 >> v1:172.21.15.22:6790/0 conn(0x7f30a813f710 legacy=0x7f30a81d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 shutdown_connections 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 >> 172.21.15.22:0/2341964341 conn(0x7f30a80b7ea0 msgr2=0x7f30a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 shutdown_connections 2024-03-25T18:25:46.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 1 -- 172.21.15.22:0/2341964341 wait complete. 2024-03-25T18:25:46.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:46.083+0000 7f30af56c640 1 librados: shutdown 2024-03-25T18:25:46.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:52.904 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:53.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.021+0000 7f7e4249a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.021+0000 7f7e4249a640 10 monclient: build_initial_monmap 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.021+0000 7f7e4249a640 10 monclient: monmap: 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:25:53.022624+0000 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:25:53.022624+0000 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:25:53.023 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:53.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 1 Processor -- start 2024-03-25T18:25:53.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 1 -- start start 2024-03-25T18:25:53.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient: init 2024-03-25T18:25:53.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3c06ab80 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3c06af60 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient(hunting): picked mon.noname-c con 0x7f7e3c0b00b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c0b0500 con 0x7f7e3c06ab80 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c13ffa0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c0af520 con 0x7f7e3c06af60 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:53.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.022+0000 7f7e4249a640 10 monclient(hunting): authenticate will time out at 7552.246539s 2024-03-25T18:25:53.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.029+0000 7f7e3bfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f7e3c06ab80 0x7f7e3c0ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:43626/0 (socket says 172.21.15.22:43626) 2024-03-25T18:25:53.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.029+0000 7f7e3bfff640 1 -- 172.21.15.22:0/3972841110 learned_addr learned my addr 172.21.15.22:0/3972841110 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:53.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.041+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573884689 0 0) 0x7f7e3c0b0500 con 0x7f7e3c06ab80 2024-03-25T18:25:53.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.041+0000 7f7e3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:53.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.041+0000 7f7e3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:53.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.041+0000 7f7e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:53.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.041+0000 7f7e3affd640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:25:53.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.041+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e24003520 con 0x7f7e3c06ab80 2024-03-25T18:25:53.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.052+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345027662 0 0) 0x7f7e3c0af520 con 0x7f7e3c06af60 2024-03-25T18:25:53.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.052+0000 7f7e3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:53.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.052+0000 7f7e3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:53.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.052+0000 7f7e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:53.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.052+0000 7f7e3affd640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:25:53.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.052+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3c0b0500 con 0x7f7e3c06af60 2024-03-25T18:25:53.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.067+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162538109 0 0) 0x7f7e3c13ffa0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.067+0000 7f7e3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.067+0000 7f7e3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.067+0000 7f7e3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.067+0000 7f7e3affd640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.067+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3c0af520 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.067+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1328583339 0 0) 0x7f7e24003520 con 0x7f7e3c06ab80 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 >> v1:172.21.15.22:6790/0 conn(0x7f7e3c06af60 legacy=0x7f7e3c13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 >> v1:172.21.15.33:6789/0 conn(0x7f7e3c0b00b0 legacy=0x7f7e3c14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 1 monclient: found mon.noname-a 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c072890 con 0x7f7e3c06ab80 2024-03-25T18:25:53.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 10 monclient: _finish_auth 0 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7e2c001690 con 0x7f7e3c06ab80 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e3affd640 10 monclient: dump: 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:53.071 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:53.072 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:53.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e4249a640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:25:53.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.068+0000 7f7e4249a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:53.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e2c0017f0 con 0x7f7e3c06ab80 2024-03-25T18:25:53.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:53.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e3affd640 1 -- 172.21.15.22:0/3972841110 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7e2c001c00 con 0x7f7e3c06ab80 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e3affd640 10 monclient: dump: 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:53.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e4249a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:53.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e4249a640 4 set_mon_vals no callback set 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e4249a640 10 monclient: shutdown 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e4249a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.081+0000 7f7e4249a640 1 -- 172.21.15.22:0/3972841110 >> v1:172.21.15.22:6789/0 conn(0x7f7e3c06ab80 legacy=0x7f7e3c0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.082+0000 7f7e4249a640 1 -- 172.21.15.22:0/3972841110 shutdown_connections 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.082+0000 7f7e4249a640 1 -- 172.21.15.22:0/3972841110 >> 172.21.15.22:0/3972841110 conn(0x7f7e3c0b4a80 msgr2=0x7f7e3c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.082+0000 7f7e4249a640 1 -- 172.21.15.22:0/3972841110 shutdown_connections 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.082+0000 7f7e4249a640 1 -- 172.21.15.22:0/3972841110 wait complete. 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 10 monclient: build_initial_monmap 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 10 monclient: monmap: 2024-03-25T18:25:53.085 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 1 librados: starting msgr at 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 1 librados: starting objecter 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 1 Processor -- start 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 1 -- start start 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 1 librados: setting wanted keys 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 1 librados: calling monclient init 2024-03-25T18:25:53.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.083+0000 7f7e4249a640 10 monclient: init 2024-03-25T18:25:53.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:53.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:53.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 10 monclient(hunting): picked mon.noname-a con 0x7f7e3c06ab80 addr v1:172.21.15.22:6789/0 2024-03-25T18:25:53.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 10 monclient(hunting): picked mon.noname-b con 0x7f7e3c0b00b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:25:53.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 10 monclient(hunting): picked mon.noname-c con 0x7f7e3c1427f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:25:53.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c145150 con 0x7f7e3c06ab80 2024-03-25T18:25:53.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c0aa320 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e3c1df830 con 0x7f7e3c1427f0 2024-03-25T18:25:53.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:53.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.084+0000 7f7e4249a640 10 monclient(hunting): authenticate will time out at 7552.308070s 2024-03-25T18:25:53.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.100+0000 7f7e3bfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f7e3c06ab80 0x7f7e3c1df120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:43668/0 (socket says 172.21.15.22:43668) 2024-03-25T18:25:53.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.100+0000 7f7e3bfff640 1 -- 172.21.15.22:0/3997623042 learned_addr learned my addr 172.21.15.22:0/3997623042 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:25:53.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.101+0000 7f7e38ff9640 10 client.?.objecter ms_handle_connect 0x7f7e3c06ab80 2024-03-25T18:25:53.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.101+0000 7f7e38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:53.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.101+0000 7f7e38ff9640 10 client.?.objecter ms_handle_connect 0x7f7e3c0b00b0 2024-03-25T18:25:53.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.101+0000 7f7e38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:53.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.110+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866944623 0 0) 0x7f7e3c0aa320 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.110+0000 7f7e38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:53.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.110+0000 7f7e38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:53.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.110+0000 7f7e38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:53.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.110+0000 7f7e38ff9640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:25:53.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.110+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e28004eb0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.112+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251367208 0 0) 0x7f7e3c145150 con 0x7f7e3c06ab80 2024-03-25T18:25:53.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.112+0000 7f7e38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:53.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.112+0000 7f7e38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:53.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.112+0000 7f7e38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:53.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.112+0000 7f7e38ff9640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:25:53.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.112+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3c0aa320 con 0x7f7e3c06ab80 2024-03-25T18:25:53.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.115+0000 7f7e38ff9640 10 client.?.objecter ms_handle_connect 0x7f7e3c1427f0 2024-03-25T18:25:53.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.115+0000 7f7e38ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:53.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.127+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3816100257 0 0) 0x7f7e28004eb0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.127+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e3c145150 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.127+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7e30001970 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.127+0000 7f7e38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.127+0000 7f7e38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.127+0000 7f7e38ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:53.129 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:53.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.127+0000 7f7e38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:53.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.129+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484661281 0 0) 0x7f7e3c1df830 con 0x7f7e3c1427f0 2024-03-25T18:25:53.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.129+0000 7f7e38ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:53.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.129+0000 7f7e38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:53.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.129+0000 7f7e38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:53.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.129+0000 7f7e38ff9640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:25:53.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.129+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e28004eb0 con 0x7f7e3c1427f0 2024-03-25T18:25:53.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3510865290 0 0) 0x7f7e3c145150 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 >> v1:172.21.15.22:6790/0 conn(0x7f7e3c1427f0 legacy=0x7f7e3c0aca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 >> v1:172.21.15.22:6789/0 conn(0x7f7e3c06ab80 legacy=0x7f7e3c1df120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 1 monclient: found mon.b 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c1441f0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e4249a640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e4249a640 10 monclient: _renew_subs 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e4249a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.132+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3c1444a0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.133+0000 7f7e4249a640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:53.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.133+0000 7f7e4249a640 10 monclient: _renew_subs 2024-03-25T18:25:53.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.133+0000 7f7e4249a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:53.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.133+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3c0ad2f0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.133+0000 7f7e4249a640 1 librados: init done 2024-03-25T18:25:53.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.134+0000 7f7e4249a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:53.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.134+0000 7f7e4249a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:53.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.134+0000 7f7e4249a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:53.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.134+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e3c0ad5a0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.419+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e30003610 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.419+0000 7f7e38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:53.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.419+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7e30003920 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.419+0000 7f7e38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:53.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.419+0000 7f7e38ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:53.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.419+0000 7f7e38ff9640 10 monclient: dump: 2024-03-25T18:25:53.421 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:25:53.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.419+0000 7f7e3a7fc640 4 set_mon_vals no callback set 2024-03-25T18:25:53.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.420+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f7e300198f0 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.421+0000 7f7e3bfff640 10 monclient: get_auth_request con 0x7f7e2806de00 auth_method 0 2024-03-25T18:25:53.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.421+0000 7f7e38ff9640 10 client.4522.objecter ms_handle_connect 0x7f7e2806de00 2024-03-25T18:25:53.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.428+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f7e30053640 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.428+0000 7f7e38ff9640 10 client.4522.objecter ms_dispatch 0x7f7e3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:53.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.428+0000 7f7e38ff9640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:53.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.428+0000 7f7e38ff9640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:53.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.429+0000 7f7e38ff9640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:25:53.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.431+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e3004ef80 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.431+0000 7f7e38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:53.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.431+0000 7f7e38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:53.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.661+0000 7f7e4249a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:53.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.661+0000 7f7e4249a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:53.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.661+0000 7f7e4249a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:25:53.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.661+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e3c143720 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.683+0000 7f7e38ff9640 1 -- 172.21.15.22:0/3997623042 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f7e30051080 con 0x7f7e3c0b00b0 2024-03-25T18:25:53.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.683+0000 7f7e38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:53.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.683+0000 7f7e38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:53.685 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:25:53.685 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:53.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 10 librados: watch_flush enter 2024-03-25T18:25:53.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 10 librados: watch_flush exit 2024-03-25T18:25:53.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:25:53.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:25:53.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 >> v1:172.21.15.33:6800/215350958 conn(0x7f7e2806de00 legacy=0x7f7e280702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:53.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 10 monclient: shutdown 2024-03-25T18:25:53.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:53.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 >> v1:172.21.15.33:6789/0 conn(0x7f7e3c0b00b0 legacy=0x7f7e3c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:53.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 shutdown_connections 2024-03-25T18:25:53.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 >> 172.21.15.22:0/3997623042 conn(0x7f7e3c0b4a80 msgr2=0x7f7e3c070510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:53.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 shutdown_connections 2024-03-25T18:25:53.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.686+0000 7f7e4249a640 1 -- 172.21.15.22:0/3997623042 wait complete. 2024-03-25T18:25:53.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:25:53.687+0000 7f7e4249a640 1 librados: shutdown 2024-03-25T18:25:53.699 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:58.463 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:01.124 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.471 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:01.133 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.501 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:01.163 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.517 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:01.133 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.527 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:01.145 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.533 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:01.195 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.542 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:01.204 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.561 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:01.179 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:58.585 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:01.247 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.506 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.619+0000 7fd3424d3640 10 monclient: get_monmap_and_config 2024-03-25T18:26:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.619+0000 7fd3424d3640 10 monclient: build_initial_monmap 2024-03-25T18:26:00.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.619+0000 7fd3424d3640 10 monclient: monmap: 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:00.620456+0000 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:00.620456+0000 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:00.621 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.619+0000 7fd3424d3640 1 Processor -- start 2024-03-25T18:26:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 1 -- start start 2024-03-25T18:26:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient: init 2024-03-25T18:26:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c0b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c0b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:00.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c14be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:00.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0af940 con 0x7fd33c0b1730 2024-03-25T18:26:00.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b0a70 con 0x7fd33c14be90 2024-03-25T18:26:00.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c0b0d50 con 0x7fd33c0b4d60 2024-03-25T18:26:00.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:00.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.620+0000 7fd3424d3640 10 monclient(hunting): authenticate will time out at 7559.844276s 2024-03-25T18:26:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.636+0000 7fd340a70640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fd33c14be90 0x7fd33c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:44800/0 (socket says 172.21.15.22:44800) 2024-03-25T18:26:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.636+0000 7fd340a70640 1 -- 172.21.15.22:0/18428991 learned_addr learned my addr 172.21.15.22:0/18428991 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.654+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545360204 0 0) 0x7fd33c0af940 con 0x7fd33c0b1730 2024-03-25T18:26:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.654+0000 7fd33affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.654+0000 7fd33affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.654+0000 7fd33affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.654+0000 7fd33affd640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:26:00.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.654+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd324003520 con 0x7fd33c0b1730 2024-03-25T18:26:00.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.666+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854915372 0 0) 0x7fd33c0b0d50 con 0x7fd33c0b4d60 2024-03-25T18:26:00.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.666+0000 7fd33affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:00.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.666+0000 7fd33affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:00.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.666+0000 7fd33affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:00.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.666+0000 7fd33affd640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:26:00.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.666+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c0af940 con 0x7fd33c0b4d60 2024-03-25T18:26:00.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.678+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199318952 0 0) 0x7fd33c0b0a70 con 0x7fd33c14be90 2024-03-25T18:26:00.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.678+0000 7fd33affd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:00.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.678+0000 7fd33affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:00.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.678+0000 7fd33affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:00.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.678+0000 7fd33affd640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:26:00.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.678+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33c0b0d50 con 0x7fd33c14be90 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566405951 0 0) 0x7fd33c0af940 con 0x7fd33c0b4d60 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 >> v1:172.21.15.33:6789/0 conn(0x7fd33c14be90 legacy=0x7fd33c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 >> v1:172.21.15.22:6789/0 conn(0x7fd33c0b1730 legacy=0x7fd33c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 1 monclient: found mon.noname-b 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c0b1030 con 0x7fd33c0b4d60 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 10 monclient: _finish_auth 0 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd32c001670 con 0x7fd33c0b4d60 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:00.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd33affd640 10 monclient: dump: 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd3424d3640 5 monclient: authenticate success, global_id 4529 2024-03-25T18:26:00.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.682+0000 7fd3424d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:00.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd32c0017d0 con 0x7fd33c0b4d60 2024-03-25T18:26:00.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd33affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:00.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd33affd640 1 -- 172.21.15.22:0/18428991 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd32c001be0 con 0x7fd33c0b4d60 2024-03-25T18:26:00.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd3424d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:00.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd3424d3640 4 set_mon_vals no callback set 2024-03-25T18:26:00.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd33affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:00.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd33affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.706+0000 7fd33affd640 10 monclient: dump: 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:00.709 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:00.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.707+0000 7fd3424d3640 10 monclient: shutdown 2024-03-25T18:26:00.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.707+0000 7fd3424d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:00.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.707+0000 7fd3424d3640 1 -- 172.21.15.22:0/18428991 >> v1:172.21.15.22:6790/0 conn(0x7fd33c0b4d60 legacy=0x7fd33c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:00.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.707+0000 7fd3424d3640 1 -- 172.21.15.22:0/18428991 shutdown_connections 2024-03-25T18:26:00.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.707+0000 7fd3424d3640 1 -- 172.21.15.22:0/18428991 >> 172.21.15.22:0/18428991 conn(0x7fd33c0617a0 msgr2=0x7fd33c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.707+0000 7fd3424d3640 1 -- 172.21.15.22:0/18428991 shutdown_connections 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.707+0000 7fd3424d3640 1 -- 172.21.15.22:0/18428991 wait complete. 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.708+0000 7fd3424d3640 10 monclient: build_initial_monmap 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.708+0000 7fd3424d3640 10 monclient: monmap: 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:00.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.708+0000 7fd3424d3640 1 librados: starting msgr at 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.708+0000 7fd3424d3640 1 librados: starting objecter 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.708+0000 7fd3424d3640 1 Processor -- start 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.709+0000 7fd3424d3640 1 -- start start 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.709+0000 7fd3424d3640 1 librados: setting wanted keys 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.709+0000 7fd3424d3640 1 librados: calling monclient init 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.709+0000 7fd3424d3640 10 monclient: init 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 10 monclient(hunting): picked mon.noname-c con 0x7fd33c0b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 10 monclient(hunting): picked mon.noname-b con 0x7fd33c0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 10 monclient(hunting): picked mon.noname-a con 0x7fd33c14be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c1408a0 con 0x7fd33c14be90 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c1327e0 con 0x7fd33c0b4d60 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33c141ea0 con 0x7fd33c0b1730 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:00.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.710+0000 7fd3424d3640 10 monclient(hunting): authenticate will time out at 7559.933869s 2024-03-25T18:26:00.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.711+0000 7fd33b7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fd33c0b4d60 0x7fd33c1319c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:44854/0 (socket says 172.21.15.22:44854) 2024-03-25T18:26:00.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.711+0000 7fd33b7fe640 1 -- 172.21.15.22:0/222929983 learned_addr learned my addr 172.21.15.22:0/222929983 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:00.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.712+0000 7fd338ff9640 10 client.?.objecter ms_handle_connect 0x7fd33c0b4d60 2024-03-25T18:26:00.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.712+0000 7fd338ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:00.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.732+0000 7fd338ff9640 10 client.?.objecter ms_handle_connect 0x7fd33c0b1730 2024-03-25T18:26:00.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.732+0000 7fd338ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.737+0000 7fd338ff9640 10 client.?.objecter ms_handle_connect 0x7fd33c14be90 2024-03-25T18:26:00.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.737+0000 7fd338ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:00.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.794+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979881612 0 0) 0x7fd33c1327e0 con 0x7fd33c0b4d60 2024-03-25T18:26:00.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.794+0000 7fd338ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:00.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.794+0000 7fd338ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:00.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.794+0000 7fd338ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:00.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.794+0000 7fd338ff9640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:26:00.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.794+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd328004eb0 con 0x7fd33c0b4d60 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.808+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 498056499 0 0) 0x7fd328004eb0 con 0x7fd33c0b4d60 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.808+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd33c1327e0 con 0x7fd33c0b4d60 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.808+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd330001990 con 0x7fd33c0b4d60 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.808+0000 7fd338ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.808+0000 7fd338ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.808+0000 7fd338ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.810 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:00.811 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:00.811 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:00.811 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:00.811 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:00.811 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:00.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.808+0000 7fd338ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:00.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.814+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128347548 0 0) 0x7fd33c1408a0 con 0x7fd33c14be90 2024-03-25T18:26:00.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.814+0000 7fd338ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:00.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.814+0000 7fd338ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:00.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.814+0000 7fd338ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:00.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.814+0000 7fd338ff9640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:26:00.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.814+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd328004eb0 con 0x7fd33c14be90 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.831+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 625449340 0 0) 0x7fd328004eb0 con 0x7fd33c14be90 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.831+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd33c1408a0 con 0x7fd33c14be90 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.831+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd334001bd0 con 0x7fd33c14be90 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.831+0000 7fd338ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.831+0000 7fd338ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.831+0000 7fd338ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:00.833 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:00.834 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:00.834 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:00.834 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2587764102 0 0) 0x7fd33c1408a0 con 0x7fd33c14be90 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 >> v1:172.21.15.22:6790/0 conn(0x7fd33c0b1730 legacy=0x7fd33c1d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 >> v1:172.21.15.33:6789/0 conn(0x7fd33c0b4d60 legacy=0x7fd33c1319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 1 monclient: found mon.a 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33c140380 con 0x7fd33c14be90 2024-03-25T18:26:00.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd338ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 10 monclient: _renew_subs 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33c137d50 con 0x7fd33c14be90 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 10 monclient: _renew_subs 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33c145390 con 0x7fd33c14be90 2024-03-25T18:26:00.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.848+0000 7fd3424d3640 1 librados: init done 2024-03-25T18:26:00.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.849+0000 7fd3424d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:00.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.849+0000 7fd3424d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:00.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.849+0000 7fd3424d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:00.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:00.849+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33c138050 con 0x7fd33c14be90 2024-03-25T18:26:01.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.136+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd334001fe0 con 0x7fd33c14be90 2024-03-25T18:26:01.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.136+0000 7fd338ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:01.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.136+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd334004d00 con 0x7fd33c14be90 2024-03-25T18:26:01.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.136+0000 7fd338ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:01.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.136+0000 7fd338ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.136+0000 7fd338ff9640 10 monclient: dump: 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.136+0000 7fd33a7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:01.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.137+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fd33401acd0 con 0x7fd33c14be90 2024-03-25T18:26:01.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.137+0000 7fd33bfff640 10 monclient: get_auth_request con 0x7fd32806dea0 auth_method 0 2024-03-25T18:26:01.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.137+0000 7fd338ff9640 10 client.4521.objecter ms_handle_connect 0x7fd32806dea0 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fd334054950 con 0x7fd33c14be90 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 10 client.4521.objecter ms_dispatch 0x7fd33c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd334050290 con 0x7fd33c14be90 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:01.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.142+0000 7fd338ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:01.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.373+0000 7fd3424d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:01.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.373+0000 7fd3424d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:01.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.373+0000 7fd3424d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:01.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.373+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd33c0627b0 con 0x7fd33c14be90 2024-03-25T18:26:01.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.421+0000 7fd338ff9640 1 -- 172.21.15.22:0/222929983 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fd334052390 con 0x7fd33c14be90 2024-03-25T18:26:01.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.421+0000 7fd338ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:01.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.421+0000 7fd338ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:01.423 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:01.423 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:01.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 10 librados: watch_flush enter 2024-03-25T18:26:01.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 10 librados: watch_flush exit 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 >> v1:172.21.15.33:6800/215350958 conn(0x7fd32806dea0 legacy=0x7fd328070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 10 monclient: shutdown 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.423+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 >> v1:172.21.15.22:6789/0 conn(0x7fd33c14be90 legacy=0x7fd33c1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.424+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 shutdown_connections 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.424+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 >> 172.21.15.22:0/222929983 conn(0x7fd33c0617a0 msgr2=0x7fd33c0b6b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.424+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 shutdown_connections 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.424+0000 7fd3424d3640 1 -- 172.21.15.22:0/222929983 wait complete. 2024-03-25T18:26:01.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:01.424+0000 7fd3424d3640 1 librados: shutdown 2024-03-25T18:26:01.437 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:08.244 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.360+0000 7f67e28db640 10 monclient: get_monmap_and_config 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.360+0000 7f67e28db640 10 monclient: build_initial_monmap 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.360+0000 7f67e28db640 10 monclient: monmap: 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:08.361964+0000 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:08.361964+0000 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:08.362 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:08.363 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:08.363 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:08.363 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.361+0000 7f67e28db640 1 Processor -- start 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.361+0000 7f67e28db640 1 -- start start 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.361+0000 7f67e28db640 10 monclient: init 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 10 monclient(hunting): picked mon.noname-a con 0x7f67dc0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 10 monclient(hunting): picked mon.noname-c con 0x7f67dc13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 10 monclient(hunting): picked mon.noname-b con 0x7f67dc1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:08.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc142520 con 0x7f67dc0a98f0 2024-03-25T18:26:08.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc0b28a0 con 0x7f67dc13f710 2024-03-25T18:26:08.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc0b2a30 con 0x7f67dc1430c0 2024-03-25T18:26:08.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:08.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.362+0000 7f67e28db640 10 monclient(hunting): authenticate will time out at 7567.586026s 2024-03-25T18:26:08.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.370+0000 7f67dbfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f67dc0a98f0 0x7f67dc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:57460/0 (socket says 172.21.15.22:57460) 2024-03-25T18:26:08.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.370+0000 7f67dbfff640 1 -- 172.21.15.22:0/1645334456 learned_addr learned my addr 172.21.15.22:0/1645334456 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.387+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383130920 0 0) 0x7f67dc142520 con 0x7f67dc0a98f0 2024-03-25T18:26:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.387+0000 7f67daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.387+0000 7f67daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.387+0000 7f67daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.387+0000 7f67daffd640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:26:08.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.387+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c4003520 con 0x7f67dc0a98f0 2024-03-25T18:26:08.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.403+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751033501 0 0) 0x7f67dc0b28a0 con 0x7f67dc13f710 2024-03-25T18:26:08.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.403+0000 7f67daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:08.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.403+0000 7f67daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:08.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.403+0000 7f67daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:08.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.403+0000 7f67daffd640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:26:08.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.403+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67dc142520 con 0x7f67dc13f710 2024-03-25T18:26:08.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.409+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100468518 0 0) 0x7f67dc0b2a30 con 0x7f67dc1430c0 2024-03-25T18:26:08.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.409+0000 7f67daffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:08.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.409+0000 7f67daffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:08.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.409+0000 7f67daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:08.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.409+0000 7f67daffd640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:26:08.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.409+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67dc0b28a0 con 0x7f67dc1430c0 2024-03-25T18:26:08.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2069555167 0 0) 0x7f67dc142520 con 0x7f67dc13f710 2024-03-25T18:26:08.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 >> v1:172.21.15.22:6790/0 conn(0x7f67dc1430c0 legacy=0x7f67dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 >> v1:172.21.15.22:6789/0 conn(0x7f67dc0a98f0 legacy=0x7f67dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:08.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 1 monclient: found mon.noname-c 2024-03-25T18:26:08.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc0b44c0 con 0x7f67dc13f710 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f67d0001670 con 0x7f67dc13f710 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67daffd640 10 monclient: dump: 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:08.414 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67e28db640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:26:08.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.411+0000 7f67e28db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.423+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67d00017d0 con 0x7f67dc13f710 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.423+0000 7f67daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.423+0000 7f67daffd640 1 -- 172.21.15.22:0/1645334456 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f67d0001be0 con 0x7f67dc13f710 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.423+0000 7f67e28db640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.423+0000 7f67e28db640 4 set_mon_vals no callback set 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:08.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67daffd640 10 monclient: dump: 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:08.426 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:08.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67e28db640 10 monclient: shutdown 2024-03-25T18:26:08.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67e28db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:08.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67e28db640 1 -- 172.21.15.22:0/1645334456 >> v1:172.21.15.33:6789/0 conn(0x7f67dc13f710 legacy=0x7f67dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67e28db640 1 -- 172.21.15.22:0/1645334456 shutdown_connections 2024-03-25T18:26:08.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67e28db640 1 -- 172.21.15.22:0/1645334456 >> 172.21.15.22:0/1645334456 conn(0x7f67dc0b7ea0 msgr2=0x7f67dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:08.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67e28db640 1 -- 172.21.15.22:0/1645334456 shutdown_connections 2024-03-25T18:26:08.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.424+0000 7f67e28db640 1 -- 172.21.15.22:0/1645334456 wait complete. 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.425+0000 7f67e28db640 10 monclient: build_initial_monmap 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.425+0000 7f67e28db640 10 monclient: monmap: 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:08.430 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.425+0000 7f67e28db640 1 librados: starting msgr at 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.425+0000 7f67e28db640 1 librados: starting objecter 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 1 Processor -- start 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 1 -- start start 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 1 librados: setting wanted keys 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 1 librados: calling monclient init 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient: init 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient(hunting): picked mon.noname-b con 0x7f67dc0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient(hunting): picked mon.noname-a con 0x7f67dc13f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient(hunting): picked mon.noname-c con 0x7f67dc1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc1d7490 con 0x7f67dc13f710 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc0abb90 con 0x7f67dc0a98f0 2024-03-25T18:26:08.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67dc1d76c0 con 0x7f67dc1430c0 2024-03-25T18:26:08.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:08.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.426+0000 7f67e28db640 10 monclient(hunting): authenticate will time out at 7567.650480s 2024-03-25T18:26:08.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.442+0000 7f67e0e78640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f67dc1430c0 0x7f67dc133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58884/0 (socket says 172.21.15.22:58884) 2024-03-25T18:26:08.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.442+0000 7f67e0e78640 1 -- 172.21.15.22:0/404621356 learned_addr learned my addr 172.21.15.22:0/404621356 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:08.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.446+0000 7f67d8ff9640 10 client.?.objecter ms_handle_connect 0x7f67dc1430c0 2024-03-25T18:26:08.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.446+0000 7f67d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:08.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.448+0000 7f67d8ff9640 10 client.?.objecter ms_handle_connect 0x7f67dc13f710 2024-03-25T18:26:08.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.448+0000 7f67d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.457+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776911509 0 0) 0x7f67dc1d76c0 con 0x7f67dc1430c0 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.457+0000 7f67d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.457+0000 7f67d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.457+0000 7f67d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.457+0000 7f67d8ff9640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.457+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c8004eb0 con 0x7f67dc1430c0 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.458+0000 7f67d8ff9640 10 client.?.objecter ms_handle_connect 0x7f67dc0a98f0 2024-03-25T18:26:08.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.458+0000 7f67d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:08.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.468+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784598936 0 0) 0x7f67dc1d7490 con 0x7f67dc13f710 2024-03-25T18:26:08.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.468+0000 7f67d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:08.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.468+0000 7f67d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:08.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.468+0000 7f67d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:08.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.468+0000 7f67d8ff9640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:26:08.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.468+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67dc1d76c0 con 0x7f67dc13f710 2024-03-25T18:26:08.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.472+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598723877 0 0) 0x7f67dc0abb90 con 0x7f67dc0a98f0 2024-03-25T18:26:08.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.472+0000 7f67d8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:08.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.472+0000 7f67d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:08.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.472+0000 7f67d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:08.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.472+0000 7f67d8ff9640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:26:08.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.472+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67dc1d7490 con 0x7f67dc0a98f0 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.477+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3811913405 0 0) 0x7f67dc1d76c0 con 0x7f67dc13f710 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.477+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67dc0abb90 con 0x7f67dc13f710 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.477+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f67d0001ae0 con 0x7f67dc13f710 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.477+0000 7f67d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.477+0000 7f67d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.477+0000 7f67d8ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:08.479 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:08.480 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:08.480 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:08.480 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:08.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.477+0000 7f67d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:08.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1525659237 0 0) 0x7f67dc0abb90 con 0x7f67dc13f710 2024-03-25T18:26:08.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 >> v1:172.21.15.22:6790/0 conn(0x7f67dc1430c0 legacy=0x7f67dc133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 >> v1:172.21.15.33:6789/0 conn(0x7f67dc0a98f0 legacy=0x7f67dc1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:08.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:08.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 1 monclient: found mon.a 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc1d82f0 con 0x7f67dc13f710 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 10 monclient: _renew_subs 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc138c20 con 0x7f67dc13f710 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 10 monclient: _renew_subs 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc1379e0 con 0x7f67dc13f710 2024-03-25T18:26:08.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.492+0000 7f67e28db640 1 librados: init done 2024-03-25T18:26:08.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.494+0000 7f67e28db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:08.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.494+0000 7f67e28db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:08.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.494+0000 7f67e28db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:08.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.494+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc0b3ef0 con 0x7f67dc13f710 2024-03-25T18:26:08.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67d0001210 con 0x7f67dc13f710 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f67d0003a40 con 0x7f67dc13f710 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67d8ff9640 10 monclient: dump: 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:08.839 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:08.840 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:08.840 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:08.840 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:08.840 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:08.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67da7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:08.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.837+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f67d0019a10 con 0x7f67dc13f710 2024-03-25T18:26:08.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.838+0000 7f67dbfff640 10 monclient: get_auth_request con 0x7f67c806de20 auth_method 0 2024-03-25T18:26:08.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.838+0000 7f67d8ff9640 10 client.4527.objecter ms_handle_connect 0x7f67c806de20 2024-03-25T18:26:08.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f67d00536a0 con 0x7f67dc13f710 2024-03-25T18:26:08.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 10 client.4527.objecter ms_dispatch 0x7f67dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:08.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:08.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:08.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-25T18:26:08.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67d004efe0 con 0x7f67dc13f710 2024-03-25T18:26:08.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:08.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:08.844+0000 7f67d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:09.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.067+0000 7f67e28db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:09.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.067+0000 7f67e28db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:09.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.067+0000 7f67e28db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:26:09.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.067+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67dc14f320 con 0x7f67dc13f710 2024-03-25T18:26:09.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.103+0000 7f67d8ff9640 1 -- 172.21.15.22:0/404621356 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f67d00510e0 con 0x7f67dc13f710 2024-03-25T18:26:09.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.103+0000 7f67d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:09.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.103+0000 7f67d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:09.105 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:09.105 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 10 librados: watch_flush enter 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 10 librados: watch_flush exit 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 10 client.4527.objecter successfully canceled tick 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 >> v1:172.21.15.33:6800/215350958 conn(0x7f67c806de20 legacy=0x7f67c80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 10 monclient: shutdown 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:09.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.105+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 >> v1:172.21.15.22:6789/0 conn(0x7f67dc13f710 legacy=0x7f67dc138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:09.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.106+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 shutdown_connections 2024-03-25T18:26:09.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.106+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 >> 172.21.15.22:0/404621356 conn(0x7f67dc0b7ea0 msgr2=0x7f67dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:09.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.106+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 shutdown_connections 2024-03-25T18:26:09.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.106+0000 7f67e28db640 1 -- 172.21.15.22:0/404621356 wait complete. 2024-03-25T18:26:09.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:09.106+0000 7f67e28db640 1 librados: shutdown 2024-03-25T18:26:09.119 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:15.927 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:16.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.042+0000 7f68865fb640 10 monclient: get_monmap_and_config 2024-03-25T18:26:16.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.042+0000 7f68865fb640 10 monclient: build_initial_monmap 2024-03-25T18:26:16.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.042+0000 7f68865fb640 10 monclient: monmap: 2024-03-25T18:26:16.044 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:16.044 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:16.044 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:16.044071+0000 2024-03-25T18:26:16.045 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:16.044071+0000 2024-03-25T18:26:16.045 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:16.045 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:16.045 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:16.045 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:16.045 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:16.045 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.043+0000 7f68865fb640 1 Processor -- start 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.043+0000 7f68865fb640 1 -- start start 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.043+0000 7f68865fb640 10 monclient: init 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 10 monclient(hunting): picked mon.noname-b con 0x7f68800705b0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 10 monclient(hunting): picked mon.noname-c con 0x7f68801402d0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 10 monclient(hunting): picked mon.noname-a con 0x7f6880149ae0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800718c0 con 0x7f6880149ae0 2024-03-25T18:26:16.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68800714c0 con 0x7f68801402d0 2024-03-25T18:26:16.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880072b50 con 0x7f68800705b0 2024-03-25T18:26:16.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:16.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.044+0000 7f68865fb640 10 monclient(hunting): authenticate will time out at 7575.267970s 2024-03-25T18:26:16.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.056+0000 7f6884b98640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6880149ae0 0x7f688014bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52958/0 (socket says 172.21.15.22:52958) 2024-03-25T18:26:16.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.056+0000 7f6884b98640 1 -- 172.21.15.22:0/4257800284 learned_addr learned my addr 172.21.15.22:0/4257800284 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:16.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.076+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365028875 0 0) 0x7f6880072b50 con 0x7f68800705b0 2024-03-25T18:26:16.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.076+0000 7f687effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:16.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.076+0000 7f687effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:16.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.076+0000 7f687effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:16.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.076+0000 7f687effd640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:26:16.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.076+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6868003520 con 0x7f68800705b0 2024-03-25T18:26:16.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.079+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566144088 0 0) 0x7f68800718c0 con 0x7f6880149ae0 2024-03-25T18:26:16.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.079+0000 7f687effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:16.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.079+0000 7f687effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:16.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.079+0000 7f687effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:16.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.079+0000 7f687effd640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:26:16.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.079+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6880072b50 con 0x7f6880149ae0 2024-03-25T18:26:16.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.081+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167252610 0 0) 0x7f68800714c0 con 0x7f68801402d0 2024-03-25T18:26:16.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.081+0000 7f687effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:16.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.081+0000 7f687effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:16.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.081+0000 7f687effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:16.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.081+0000 7f687effd640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:26:16.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.081+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68800718c0 con 0x7f68801402d0 2024-03-25T18:26:16.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.086+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2713993782 0 0) 0x7f6880072b50 con 0x7f6880149ae0 2024-03-25T18:26:16.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.086+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 >> v1:172.21.15.22:6790/0 conn(0x7f68800705b0 legacy=0x7f688013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.086+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 >> v1:172.21.15.33:6789/0 conn(0x7f68801402d0 legacy=0x7f6880140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 1 monclient: found mon.noname-a 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688013dd10 con 0x7f6880149ae0 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f68780019f0 con 0x7f6880149ae0 2024-03-25T18:26:16.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f687effd640 10 monclient: dump: 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:16.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f68865fb640 5 monclient: authenticate success, global_id 4530 2024-03-25T18:26:16.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.087+0000 7f68865fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:16.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6878001b50 con 0x7f6880149ae0 2024-03-25T18:26:16.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f687effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:16.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f687effd640 1 -- 172.21.15.22:0/4257800284 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6878001f60 con 0x7f6880149ae0 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f687effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f687effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f687effd640 10 monclient: dump: 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:16.094 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:16.095 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:16.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f68865fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:16.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.092+0000 7f68865fb640 4 set_mon_vals no callback set 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.093+0000 7f68865fb640 10 monclient: shutdown 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.093+0000 7f68865fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.093+0000 7f68865fb640 1 -- 172.21.15.22:0/4257800284 >> v1:172.21.15.22:6789/0 conn(0x7f6880149ae0 legacy=0x7f688014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.093+0000 7f68865fb640 1 -- 172.21.15.22:0/4257800284 shutdown_connections 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.093+0000 7f68865fb640 1 -- 172.21.15.22:0/4257800284 >> 172.21.15.22:0/4257800284 conn(0x7f68800b5d10 msgr2=0x7f68800b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.093+0000 7f68865fb640 1 -- 172.21.15.22:0/4257800284 shutdown_connections 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.093+0000 7f68865fb640 1 -- 172.21.15.22:0/4257800284 wait complete. 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.094+0000 7f68865fb640 10 monclient: build_initial_monmap 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.094+0000 7f68865fb640 10 monclient: monmap: 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:16.097 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.095+0000 7f68865fb640 1 librados: starting msgr at 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.095+0000 7f68865fb640 1 librados: starting objecter 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.095+0000 7f68865fb640 1 Processor -- start 2024-03-25T18:26:16.098 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.095+0000 7f68865fb640 1 -- start start 2024-03-25T18:26:16.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.095+0000 7f68865fb640 1 librados: setting wanted keys 2024-03-25T18:26:16.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.095+0000 7f68865fb640 1 librados: calling monclient init 2024-03-25T18:26:16.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.095+0000 7f68865fb640 10 monclient: init 2024-03-25T18:26:16.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:16.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:16.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 10 monclient(hunting): picked mon.noname-b con 0x7f68800705b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:16.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 10 monclient(hunting): picked mon.noname-a con 0x7f68801402d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:16.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 10 monclient(hunting): picked mon.noname-c con 0x7f6880149ae0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:16.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880135910 con 0x7f68801402d0 2024-03-25T18:26:16.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6880141870 con 0x7f68800705b0 2024-03-25T18:26:16.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68801443d0 con 0x7f6880149ae0 2024-03-25T18:26:16.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:16.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.096+0000 7f68865fb640 10 monclient(hunting): authenticate will time out at 7575.320556s 2024-03-25T18:26:16.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.111+0000 7f687ffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f68800705b0 0x7f68801422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:60536/0 (socket says 172.21.15.22:60536) 2024-03-25T18:26:16.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.111+0000 7f687ffff640 1 -- 172.21.15.22:0/2410046670 learned_addr learned my addr 172.21.15.22:0/2410046670 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:16.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.113+0000 7f687cff9640 10 client.?.objecter ms_handle_connect 0x7f6880149ae0 2024-03-25T18:26:16.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.113+0000 7f687cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:16.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.113+0000 7f687cff9640 10 client.?.objecter ms_handle_connect 0x7f68800705b0 2024-03-25T18:26:16.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.113+0000 7f687cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.115+0000 7f687cff9640 10 client.?.objecter ms_handle_connect 0x7f68801402d0 2024-03-25T18:26:16.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.115+0000 7f687cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.120+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709948655 0 0) 0x7f6880141870 con 0x7f68800705b0 2024-03-25T18:26:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.120+0000 7f687cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.120+0000 7f687cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.120+0000 7f687cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.120+0000 7f687cff9640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:26:16.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.120+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c004eb0 con 0x7f68800705b0 2024-03-25T18:26:16.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.127+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058059796 0 0) 0x7f68801443d0 con 0x7f6880149ae0 2024-03-25T18:26:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.127+0000 7f687cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.127+0000 7f687cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.127+0000 7f687cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.127+0000 7f687cff9640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:26:16.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.127+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6880141870 con 0x7f6880149ae0 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.128+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3414083532 0 0) 0x7f686c004eb0 con 0x7f68800705b0 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.128+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68801443d0 con 0x7f68800705b0 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.128+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6870001990 con 0x7f68800705b0 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.128+0000 7f687cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.128+0000 7f687cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.128+0000 7f687cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.130 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.131 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:16.131 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:16.131 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:16.131 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:16.131 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:16.131 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:16.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.128+0000 7f687cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.132+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055119185 0 0) 0x7f6880135910 con 0x7f68801402d0 2024-03-25T18:26:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.132+0000 7f687cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.132+0000 7f687cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.132+0000 7f687cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.132+0000 7f687cff9640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:26:16.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.132+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f686c004eb0 con 0x7f68801402d0 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.139+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 85642242 0 0) 0x7f68801443d0 con 0x7f68800705b0 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 >> v1:172.21.15.22:6790/0 conn(0x7f6880149ae0 legacy=0x7f68801d5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 >> v1:172.21.15.22:6789/0 conn(0x7f68801402d0 legacy=0x7f6880133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 1 monclient: found mon.b 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6880144560 con 0x7f68800705b0 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:16.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f687cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 10 monclient: _renew_subs 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6880142cb0 con 0x7f68800705b0 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 10 monclient: _renew_subs 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6880134770 con 0x7f68800705b0 2024-03-25T18:26:16.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.140+0000 7f68865fb640 1 librados: init done 2024-03-25T18:26:16.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.141+0000 7f68865fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:16.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.141+0000 7f68865fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:16.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.141+0000 7f68865fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:16.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.141+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6880142a60 con 0x7f68800705b0 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.434+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6870003610 con 0x7f68800705b0 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.434+0000 7f687cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.434+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f68700039e0 con 0x7f68800705b0 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.434+0000 7f687cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.434+0000 7f687cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.434+0000 7f687cff9640 10 monclient: dump: 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:16.436 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:16.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.434+0000 7f687e7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:16.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.435+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f6870003c00 con 0x7f68800705b0 2024-03-25T18:26:16.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.436+0000 7f687f7fe640 10 monclient: get_auth_request con 0x7f686c06de00 auth_method 0 2024-03-25T18:26:16.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.436+0000 7f687cff9640 10 client.4540.objecter ms_handle_connect 0x7f686c06de00 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f68700547b0 con 0x7f68800705b0 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 10 client.4540.objecter ms_dispatch 0x7f6880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68700502c0 con 0x7f68800705b0 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:16.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.441+0000 7f687cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:16.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.701+0000 7f68865fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:16.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.701+0000 7f68865fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:16.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.701+0000 7f68865fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:16.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.701+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68800627b0 con 0x7f68800705b0 2024-03-25T18:26:16.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.723+0000 7f687cff9640 1 -- 172.21.15.22:0/2410046670 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f68700521f0 con 0x7f68800705b0 2024-03-25T18:26:16.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.723+0000 7f687cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:16.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.723+0000 7f687cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:16.725 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:16.725 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:16.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 10 librados: watch_flush enter 2024-03-25T18:26:16.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 10 librados: watch_flush exit 2024-03-25T18:26:16.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-25T18:26:16.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 10 client.4540.objecter successfully canceled tick 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 >> v1:172.21.15.33:6800/215350958 conn(0x7f686c06de00 legacy=0x7f686c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 10 monclient: shutdown 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 >> v1:172.21.15.33:6789/0 conn(0x7f68800705b0 legacy=0x7f68801422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.725+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 shutdown_connections 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.726+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 >> 172.21.15.22:0/2410046670 conn(0x7f68800b5d10 msgr2=0x7f688013c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.726+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 shutdown_connections 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.726+0000 7f68865fb640 1 -- 172.21.15.22:0/2410046670 wait complete. 2024-03-25T18:26:16.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:16.726+0000 7f68865fb640 1 librados: shutdown 2024-03-25T18:26:16.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:18.545 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:21.207 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.552 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:21.214 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.581 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:21.243 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.602 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:21.220 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.608 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:21.226 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.613 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:21.274 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.623 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:21.284 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.642 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:21.260 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:18.660 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:21.321 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:23.546 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:23.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.662+0000 7f268876d640 10 monclient: get_monmap_and_config 2024-03-25T18:26:23.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.662+0000 7f268876d640 10 monclient: build_initial_monmap 2024-03-25T18:26:23.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.662+0000 7f268876d640 10 monclient: monmap: 2024-03-25T18:26:23.662 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:23.662466+0000 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:23.662466+0000 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:23.663 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:23.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.662+0000 7f268876d640 1 Processor -- start 2024-03-25T18:26:23.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.662+0000 7f268876d640 1 -- start start 2024-03-25T18:26:23.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.662+0000 7f268876d640 10 monclient: init 2024-03-25T18:26:23.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:23.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 10 monclient(hunting): picked mon.noname-a con 0x7f26800b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 10 monclient(hunting): picked mon.noname-c con 0x7f26800b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 10 monclient(hunting): picked mon.noname-b con 0x7f268014be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26800af940 con 0x7f26800b4d60 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26800b0a70 con 0x7f26800b1730 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26800b0d50 con 0x7f268014be90 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:23.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.663+0000 7f268876d640 10 monclient(hunting): authenticate will time out at 7582.886520s 2024-03-25T18:26:23.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.672+0000 7f2685d08640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f26800b1730 0x7f26800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:55126/0 (socket says 172.21.15.22:55126) 2024-03-25T18:26:23.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.672+0000 7f2685d08640 1 -- 172.21.15.22:0/619546643 learned_addr learned my addr 172.21.15.22:0/619546643 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:23.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.690+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115508639 0 0) 0x7f26800b0a70 con 0x7f26800b1730 2024-03-25T18:26:23.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.691+0000 7f2685507640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.691+0000 7f2685507640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.691+0000 7f2685507640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.691+0000 7f2685507640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:26:23.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.691+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f266c003520 con 0x7f26800b1730 2024-03-25T18:26:23.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.695+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577828002 0 0) 0x7f26800af940 con 0x7f26800b4d60 2024-03-25T18:26:23.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.695+0000 7f2685507640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.695+0000 7f2685507640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.695+0000 7f2685507640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.695+0000 7f2685507640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:26:23.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.695+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26800b0a70 con 0x7f26800b4d60 2024-03-25T18:26:23.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900733893 0 0) 0x7f26800b0d50 con 0x7f268014be90 2024-03-25T18:26:23.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:26:23.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26800af940 con 0x7f268014be90 2024-03-25T18:26:23.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2050545543 0 0) 0x7f266c003520 con 0x7f26800b1730 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 >> v1:172.21.15.22:6790/0 conn(0x7f268014be90 legacy=0x7f268014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 >> v1:172.21.15.22:6789/0 conn(0x7f26800b4d60 legacy=0x7f268013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient: _finish_hunting 0 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 monclient: found mon.noname-c 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26800b1030 con 0x7f26800b1730 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient: _finish_auth 0 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient: _check_auth_tickets 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2670001690 con 0x7f26800b1730 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f2685507640 10 monclient: dump: 2024-03-25T18:26:23.707 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f268876d640 5 monclient: authenticate success, global_id 4543 2024-03-25T18:26:23.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.705+0000 7f268876d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26700017f0 con 0x7f26800b1730 2024-03-25T18:26:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f2685507640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f2685507640 1 -- 172.21.15.22:0/619546643 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2670001c00 con 0x7f26800b1730 2024-03-25T18:26:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f268876d640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f268876d640 4 set_mon_vals no callback set 2024-03-25T18:26:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f2685507640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f2685507640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.737+0000 7f2685507640 10 monclient: dump: 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:23.739 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.738+0000 7f268876d640 10 monclient: shutdown 2024-03-25T18:26:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.738+0000 7f268876d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.738+0000 7f268876d640 1 -- 172.21.15.22:0/619546643 >> v1:172.21.15.33:6789/0 conn(0x7f26800b1730 legacy=0x7f26800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.738+0000 7f268876d640 1 -- 172.21.15.22:0/619546643 shutdown_connections 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.738+0000 7f268876d640 1 -- 172.21.15.22:0/619546643 >> 172.21.15.22:0/619546643 conn(0x7f26800617a0 msgr2=0x7f2680061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.738+0000 7f268876d640 1 -- 172.21.15.22:0/619546643 shutdown_connections 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.738+0000 7f268876d640 1 -- 172.21.15.22:0/619546643 wait complete. 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.739+0000 7f268876d640 10 monclient: build_initial_monmap 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.739+0000 7f268876d640 10 monclient: monmap: 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:23.741 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.739+0000 7f268876d640 1 librados: starting msgr at 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.739+0000 7f268876d640 1 librados: starting objecter 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.740+0000 7f268876d640 1 Processor -- start 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.740+0000 7f268876d640 1 -- start start 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.740+0000 7f268876d640 1 librados: setting wanted keys 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.740+0000 7f268876d640 1 librados: calling monclient init 2024-03-25T18:26:23.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.740+0000 7f268876d640 10 monclient: init 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.740+0000 7f268876d640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.740+0000 7f268876d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 10 monclient(hunting): picked mon.noname-a con 0x7f26800b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 10 monclient(hunting): picked mon.noname-c con 0x7f26800b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 10 monclient(hunting): picked mon.noname-b con 0x7f268014be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268013bd10 con 0x7f26800b1730 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2680141c30 con 0x7f268014be90 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2680141f10 con 0x7f26800b4d60 2024-03-25T18:26:23.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:23.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.741+0000 7f268876d640 10 monclient(hunting): authenticate will time out at 7582.963810s 2024-03-25T18:26:23.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.743+0000 7f2686509640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f26800b1730 0x7f26801341f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48214/0 (socket says 172.21.15.22:48214) 2024-03-25T18:26:23.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.743+0000 7f2686509640 1 -- 172.21.15.22:0/521436959 learned_addr learned my addr 172.21.15.22:0/521436959 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:23.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.750+0000 7f2676ffd640 10 client.?.objecter ms_handle_connect 0x7f26800b1730 2024-03-25T18:26:23.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.750+0000 7f2676ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:23.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.752+0000 7f2676ffd640 10 client.?.objecter ms_handle_connect 0x7f268014be90 2024-03-25T18:26:23.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.752+0000 7f2676ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:23.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.758+0000 7f2676ffd640 10 client.?.objecter ms_handle_connect 0x7f26800b4d60 2024-03-25T18:26:23.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.758+0000 7f2676ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:23.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.758+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088643124 0 0) 0x7f2680141c30 con 0x7f268014be90 2024-03-25T18:26:23.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.759+0000 7f2676ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.759+0000 7f2676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.759+0000 7f2676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.759+0000 7f2676ffd640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:26:23.759 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.759+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2664004eb0 con 0x7f268014be90 2024-03-25T18:26:23.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.769+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283691111 0 0) 0x7f268013bd10 con 0x7f26800b1730 2024-03-25T18:26:23.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.769+0000 7f2676ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.769+0000 7f2676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.769+0000 7f2676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.769+0000 7f2676ffd640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:26:23.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.769+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2680141c30 con 0x7f26800b1730 2024-03-25T18:26:23.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.770+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226319521 0 0) 0x7f2680141f10 con 0x7f26800b4d60 2024-03-25T18:26:23.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.770+0000 7f2676ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:23.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.770+0000 7f2676ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:23.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.770+0000 7f2676ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:23.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.770+0000 7f2676ffd640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:26:23.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.770+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f268013bd10 con 0x7f26800b4d60 2024-03-25T18:26:23.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.772+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3385053520 0 0) 0x7f2664004eb0 con 0x7f268014be90 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.772+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2680141f10 con 0x7f268014be90 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.772+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2678001bd0 con 0x7f268014be90 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.772+0000 7f2676ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.772+0000 7f2676ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.772+0000 7f2676ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:23.773 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:23.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.772+0000 7f2676ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.780+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2570053667 0 0) 0x7f268013bd10 con 0x7f26800b4d60 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.780+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2664004eb0 con 0x7f26800b4d60 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.780+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2670001b00 con 0x7f26800b4d60 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.780+0000 7f2676ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.780+0000 7f2676ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.780+0000 7f2676ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:23.781 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:23.782 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:23.782 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:23.782 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:23.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1575697904 0 0) 0x7f2680141f10 con 0x7f268014be90 2024-03-25T18:26:23.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 >> v1:172.21.15.22:6790/0 conn(0x7f26800b4d60 legacy=0x7f2680142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 >> v1:172.21.15.22:6789/0 conn(0x7f26800b1730 legacy=0x7f26801341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:23.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:23.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 1 monclient: found mon.b 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680140810 con 0x7f268014be90 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f2676ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 10 monclient: _renew_subs 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2680140d30 con 0x7f268014be90 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 10 monclient: _renew_subs 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2680140560 con 0x7f268014be90 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.781+0000 7f268876d640 1 librados: init done 2024-03-25T18:26:23.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.782+0000 7f268876d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:23.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.782+0000 7f268876d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:23.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.782+0000 7f268876d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:23.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:23.782+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26800627b0 con 0x7f268014be90 2024-03-25T18:26:24.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.066+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2678002060 con 0x7f268014be90 2024-03-25T18:26:24.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.066+0000 7f2676ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:24.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.066+0000 7f2684d06640 4 set_mon_vals no callback set 2024-03-25T18:26:24.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.066+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2678003b30 con 0x7f268014be90 2024-03-25T18:26:24.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.066+0000 7f2676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:24.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.066+0000 7f2676ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:24.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.066+0000 7f2676ffd640 10 monclient: dump: 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:24.068 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:24.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.067+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f2678003d50 con 0x7f268014be90 2024-03-25T18:26:24.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.067+0000 7f2686509640 10 monclient: get_auth_request con 0x7f266406df40 auth_method 0 2024-03-25T18:26:24.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.068+0000 7f2676ffd640 10 client.4546.objecter ms_handle_connect 0x7f266406df40 2024-03-25T18:26:24.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.073+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f2678053830 con 0x7f268014be90 2024-03-25T18:26:24.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.073+0000 7f2676ffd640 10 client.4546.objecter ms_dispatch 0x7f2680000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:24.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.073+0000 7f2676ffd640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:24.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.073+0000 7f2676ffd640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:24.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.073+0000 7f2676ffd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:26:24.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.074+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f267804f170 con 0x7f268014be90 2024-03-25T18:26:24.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.074+0000 7f2676ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:24.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.074+0000 7f2676ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:24.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.347+0000 7f268876d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:24.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.347+0000 7f268876d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:24.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.347+0000 7f268876d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:24.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.347+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26801409a0 con 0x7f268014be90 2024-03-25T18:26:24.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.371+0000 7f2676ffd640 1 -- 172.21.15.22:0/521436959 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f2678051270 con 0x7f268014be90 2024-03-25T18:26:24.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.371+0000 7f2676ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:24.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.371+0000 7f2676ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:24.372 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:24.372 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:24.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 10 librados: watch_flush enter 2024-03-25T18:26:24.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 10 librados: watch_flush exit 2024-03-25T18:26:24.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:26:24.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 >> v1:172.21.15.33:6800/215350958 conn(0x7f266406df40 legacy=0x7f2664070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 10 monclient: shutdown 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 >> v1:172.21.15.33:6789/0 conn(0x7f268014be90 legacy=0x7f26801e35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 shutdown_connections 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 >> 172.21.15.22:0/521436959 conn(0x7f26800617a0 msgr2=0x7f26800b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 shutdown_connections 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 1 -- 172.21.15.22:0/521436959 wait complete. 2024-03-25T18:26:24.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:24.373+0000 7f268876d640 1 librados: shutdown 2024-03-25T18:26:24.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:31.194 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:31.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.307+0000 7f3599dcf640 10 monclient: get_monmap_and_config 2024-03-25T18:26:31.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.307+0000 7f3599dcf640 10 monclient: build_initial_monmap 2024-03-25T18:26:31.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.307+0000 7f3599dcf640 10 monclient: monmap: 2024-03-25T18:26:31.308 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:31.308 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:31.308 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:31.308133+0000 2024-03-25T18:26:31.309 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:31.308133+0000 2024-03-25T18:26:31.309 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:31.309 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:31.309 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:31.309 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:31.309 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:31.309 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:31.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.308+0000 7f3599dcf640 1 Processor -- start 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.308+0000 7f3599dcf640 1 -- start start 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.308+0000 7f3599dcf640 10 monclient: init 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f35940b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f35940b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f359414be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940af940 con 0x7f35940b4d60 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940b0a70 con 0x7f35940b1730 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940b0d50 con 0x7f359414be90 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:31.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.309+0000 7f3599dcf640 10 monclient(hunting): authenticate will time out at 7590.532181s 2024-03-25T18:26:31.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.318+0000 7f3592ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f35940b1730 0x7f35940b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:59668/0 (socket says 172.21.15.22:59668) 2024-03-25T18:26:31.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.318+0000 7f3592ffd640 1 -- 172.21.15.22:0/1404662954 learned_addr learned my addr 172.21.15.22:0/1404662954 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:31.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.330+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138440024 0 0) 0x7f35940b0a70 con 0x7f35940b1730 2024-03-25T18:26:31.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.330+0000 7f35927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:31.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.330+0000 7f35927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:31.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.330+0000 7f35927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:31.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.330+0000 7f35927fc640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:26:31.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.330+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f357c003520 con 0x7f35940b1730 2024-03-25T18:26:31.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 618628874 0 0) 0x7f357c003520 con 0x7f35940b1730 2024-03-25T18:26:31.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 >> v1:172.21.15.22:6790/0 conn(0x7f359414be90 legacy=0x7f359414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 >> v1:172.21.15.22:6789/0 conn(0x7f35940b4d60 legacy=0x7f359413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:31.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 1 monclient: found mon.noname-c 2024-03-25T18:26:31.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:31.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940b1030 con 0x7f35940b1730 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3588001690 con 0x7f35940b1730 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f35927fc640 10 monclient: dump: 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:31.339 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:31.340 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:31.340 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:31.340 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:31.340 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:31.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f3599dcf640 5 monclient: authenticate success, global_id 4549 2024-03-25T18:26:31.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.337+0000 7f3599dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:31.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35880017f0 con 0x7f35940b1730 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f35927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f35927fc640 1 -- 172.21.15.22:0/1404662954 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f3588001c40 con 0x7f35940b1730 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f35927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f35927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f35927fc640 10 monclient: dump: 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:31.349 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:31.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f3599dcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:31.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.348+0000 7f3599dcf640 4 set_mon_vals no callback set 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 10 monclient: shutdown 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 1 -- 172.21.15.22:0/1404662954 >> v1:172.21.15.33:6789/0 conn(0x7f35940b1730 legacy=0x7f35940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 1 -- 172.21.15.22:0/1404662954 shutdown_connections 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 1 -- 172.21.15.22:0/1404662954 >> 172.21.15.22:0/1404662954 conn(0x7f35940617a0 msgr2=0x7f3594061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 1 -- 172.21.15.22:0/1404662954 shutdown_connections 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 1 -- 172.21.15.22:0/1404662954 wait complete. 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 10 monclient: build_initial_monmap 2024-03-25T18:26:31.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.349+0000 7f3599dcf640 10 monclient: monmap: 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.350+0000 7f3599dcf640 1 librados: starting msgr at 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.350+0000 7f3599dcf640 1 librados: starting objecter 2024-03-25T18:26:31.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.350+0000 7f3599dcf640 1 Processor -- start 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.350+0000 7f3599dcf640 1 -- start start 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.350+0000 7f3599dcf640 1 librados: setting wanted keys 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.350+0000 7f3599dcf640 1 librados: calling monclient init 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.350+0000 7f3599dcf640 10 monclient: init 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f35940b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f35940b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f359414be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3594142050 con 0x7f35940b1730 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35941dac50 con 0x7f359414be90 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3594140d00 con 0x7f35940b4d60 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:31.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.351+0000 7f3599dcf640 10 monclient(hunting): authenticate will time out at 7590.574459s 2024-03-25T18:26:31.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.369+0000 7f3592ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f35940b4d60 0x7f35941db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35068/0 (socket says 172.21.15.22:35068) 2024-03-25T18:26:31.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.369+0000 7f3592ffd640 1 -- 172.21.15.22:0/380087029 learned_addr learned my addr 172.21.15.22:0/380087029 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:31.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.371+0000 7f357bfff640 10 client.?.objecter ms_handle_connect 0x7f35940b1730 2024-03-25T18:26:31.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.371+0000 7f357bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:31.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.371+0000 7f357bfff640 10 client.?.objecter ms_handle_connect 0x7f359414be90 2024-03-25T18:26:31.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.371+0000 7f357bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:31.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.374+0000 7f357bfff640 10 client.?.objecter ms_handle_connect 0x7f35940b4d60 2024-03-25T18:26:31.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.374+0000 7f357bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023074032 0 0) 0x7f3594140d00 con 0x7f35940b4d60 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3580004eb0 con 0x7f35940b4d60 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355809149 0 0) 0x7f3594142050 con 0x7f35940b1730 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:26:31.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3594140d00 con 0x7f35940b1730 2024-03-25T18:26:31.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263239803 0 0) 0x7f35941dac50 con 0x7f359414be90 2024-03-25T18:26:31.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:31.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:31.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:31.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:26:31.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.384+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3594142050 con 0x7f359414be90 2024-03-25T18:26:31.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.397+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064933399 0 0) 0x7f3594140d00 con 0x7f35940b1730 2024-03-25T18:26:31.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.397+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35941dac50 con 0x7f35940b1730 2024-03-25T18:26:31.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.397+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f35840018a0 con 0x7f35940b1730 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.397+0000 7f357bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.397+0000 7f357bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.397+0000 7f357bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:31.398 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:31.399 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:31.399 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.397+0000 7f357bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:31.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.402+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686365125 0 0) 0x7f3594142050 con 0x7f359414be90 2024-03-25T18:26:31.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.402+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3594140d00 con 0x7f359414be90 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.402+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f358c001a90 con 0x7f359414be90 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.402+0000 7f357bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.402+0000 7f357bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.402+0000 7f357bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:31.403 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:31.404 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:31.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2602494900 0 0) 0x7f3594140d00 con 0x7f359414be90 2024-03-25T18:26:31.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 >> v1:172.21.15.22:6790/0 conn(0x7f35940b4d60 legacy=0x7f35941db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 >> v1:172.21.15.22:6789/0 conn(0x7f35940b1730 legacy=0x7f35941405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:31.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 1 monclient: found mon.b 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35941423e0 con 0x7f359414be90 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f357bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 10 monclient: _renew_subs 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359413b470 con 0x7f359414be90 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 10 monclient: _renew_subs 2024-03-25T18:26:31.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:31.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35941dba40 con 0x7f359414be90 2024-03-25T18:26:31.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.407+0000 7f3599dcf640 1 librados: init done 2024-03-25T18:26:31.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.408+0000 7f3599dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:31.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.408+0000 7f3599dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:31.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.408+0000 7f3599dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:31.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.408+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35941401f0 con 0x7f359414be90 2024-03-25T18:26:31.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.699+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f358c001fa0 con 0x7f359414be90 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.699+0000 7f357bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.699+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f358c003b10 con 0x7f359414be90 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.699+0000 7f357bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.699+0000 7f357bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.699+0000 7f357bfff640 10 monclient: dump: 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:31.703 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:31.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.699+0000 7f3591ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:31.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.700+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f358c003d30 con 0x7f359414be90 2024-03-25T18:26:31.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.701+0000 7f35937fe640 10 monclient: get_auth_request con 0x7f358006dfd0 auth_method 0 2024-03-25T18:26:31.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.701+0000 7f357bfff640 10 client.4552.objecter ms_handle_connect 0x7f358006dfd0 2024-03-25T18:26:31.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f358c0536f0 con 0x7f359414be90 2024-03-25T18:26:31.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 10 client.4552.objecter ms_dispatch 0x7f3594000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:26:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f358c04f200 con 0x7f359414be90 2024-03-25T18:26:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:31.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.706+0000 7f357bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:31.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.969+0000 7f3599dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:31.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.970+0000 7f3599dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:31.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.970+0000 7f3599dcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:31.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.970+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3594140e90 con 0x7f359414be90 2024-03-25T18:26:31.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.992+0000 7f357bfff640 1 -- 172.21.15.22:0/380087029 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f358c051130 con 0x7f359414be90 2024-03-25T18:26:31.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.992+0000 7f357bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:31.992 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.992+0000 7f357bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:31.992 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:31.993 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 10 librados: watch_flush enter 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 10 librados: watch_flush exit 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 >> v1:172.21.15.33:6800/215350958 conn(0x7f358006dfd0 legacy=0x7f3580070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 10 monclient: shutdown 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:31.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 >> v1:172.21.15.33:6789/0 conn(0x7f359414be90 legacy=0x7f35941e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 shutdown_connections 2024-03-25T18:26:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.994+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 >> 172.21.15.22:0/380087029 conn(0x7f35940617a0 msgr2=0x7f35940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.995+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 shutdown_connections 2024-03-25T18:26:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.995+0000 7f3599dcf640 1 -- 172.21.15.22:0/380087029 wait complete. 2024-03-25T18:26:31.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:31.995+0000 7f3599dcf640 1 librados: shutdown 2024-03-25T18:26:32.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:38.597 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:41.259 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.608 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:41.270 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.646 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:07:41.308 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.651 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:41.269 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.653 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:41.315 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.658 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:41.276 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.660 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:41.322 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.682 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:07:41.300 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.691 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:07:41.353 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:38.812 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:38.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.926+0000 7f520470e640 10 monclient: get_monmap_and_config 2024-03-25T18:26:38.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.926+0000 7f520470e640 10 monclient: build_initial_monmap 2024-03-25T18:26:38.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.927+0000 7f520470e640 10 monclient: monmap: 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:38.927315+0000 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:38.927315+0000 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:38.929 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:38.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.927+0000 7f520470e640 1 Processor -- start 2024-03-25T18:26:38.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.927+0000 7f520470e640 1 -- start start 2024-03-25T18:26:38.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.927+0000 7f520470e640 10 monclient: init 2024-03-25T18:26:38.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:38.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:38.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 10 monclient(hunting): picked mon.noname-a con 0x7f51fc0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:38.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 10 monclient(hunting): picked mon.noname-b con 0x7f51fc13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:38.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 10 monclient(hunting): picked mon.noname-c con 0x7f51fc1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:38.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc142520 con 0x7f51fc0a98f0 2024-03-25T18:26:38.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc0b28a0 con 0x7f51fc1430c0 2024-03-25T18:26:38.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc0b2a30 con 0x7f51fc13f710 2024-03-25T18:26:38.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:38.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.928+0000 7f520470e640 10 monclient(hunting): authenticate will time out at 7598.151369s 2024-03-25T18:26:38.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.942+0000 7f5202cab640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f51fc1430c0 0x7f51fc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:36042/0 (socket says 172.21.15.22:36042) 2024-03-25T18:26:38.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.942+0000 7f5202cab640 1 -- 172.21.15.22:0/4123465492 learned_addr learned my addr 172.21.15.22:0/4123465492 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:38.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.959+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734773303 0 0) 0x7f51fc0b28a0 con 0x7f51fc1430c0 2024-03-25T18:26:38.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.959+0000 7f52014a8640 10 monclient(hunting): _negotiate 2024-03-25T18:26:38.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.959+0000 7f52014a8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:38.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.959+0000 7f52014a8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:38.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.959+0000 7f52014a8640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:26:38.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.959+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e8003520 con 0x7f51fc1430c0 2024-03-25T18:26:38.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3937646101 0 0) 0x7f51e8003520 con 0x7f51fc1430c0 2024-03-25T18:26:38.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 >> v1:172.21.15.22:6790/0 conn(0x7f51fc13f710 legacy=0x7f51fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 >> v1:172.21.15.22:6789/0 conn(0x7f51fc0a98f0 legacy=0x7f51fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 10 monclient: _finish_hunting 0 2024-03-25T18:26:38.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 1 monclient: found mon.noname-c 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc0b44c0 con 0x7f51fc1430c0 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 10 monclient: _finish_auth 0 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 10 monclient: _check_auth_tickets 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f51f4001a10 con 0x7f51fc1430c0 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f520470e640 5 monclient: authenticate success, global_id 4555 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f520470e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f52014a8640 10 monclient: dump: 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:38.972 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:38.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.970+0000 7f520470e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:38.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51f4001b70 con 0x7f51fc1430c0 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f52014a8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f52014a8640 1 -- 172.21.15.22:0/4123465492 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f51f4001fc0 con 0x7f51fc1430c0 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f520470e640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f520470e640 4 set_mon_vals no callback set 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f52014a8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f52014a8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.985+0000 7f52014a8640 10 monclient: dump: 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:38.986 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:38.987 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:38.987 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:38.987 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:38.987 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:38.987 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:38.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.986+0000 7f520470e640 10 monclient: shutdown 2024-03-25T18:26:38.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.986+0000 7f520470e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:38.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.986+0000 7f520470e640 1 -- 172.21.15.22:0/4123465492 >> v1:172.21.15.33:6789/0 conn(0x7f51fc1430c0 legacy=0x7f51fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:38.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.986+0000 7f520470e640 1 -- 172.21.15.22:0/4123465492 shutdown_connections 2024-03-25T18:26:38.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.986+0000 7f520470e640 1 -- 172.21.15.22:0/4123465492 >> 172.21.15.22:0/4123465492 conn(0x7f51fc0b7ea0 msgr2=0x7f51fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:38.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.986+0000 7f520470e640 1 -- 172.21.15.22:0/4123465492 shutdown_connections 2024-03-25T18:26:38.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.986+0000 7f520470e640 1 -- 172.21.15.22:0/4123465492 wait complete. 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.987+0000 7f520470e640 10 monclient: build_initial_monmap 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.987+0000 7f520470e640 10 monclient: monmap: 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:38.989 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.988+0000 7f520470e640 1 librados: starting msgr at 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.988+0000 7f520470e640 1 librados: starting objecter 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.988+0000 7f520470e640 1 Processor -- start 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.988+0000 7f520470e640 1 -- start start 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.988+0000 7f520470e640 1 librados: setting wanted keys 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.988+0000 7f520470e640 1 librados: calling monclient init 2024-03-25T18:26:38.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.988+0000 7f520470e640 10 monclient: init 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 10 monclient(hunting): picked mon.noname-b con 0x7f51fc0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 10 monclient(hunting): picked mon.noname-c con 0x7f51fc13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 10 monclient(hunting): picked mon.noname-a con 0x7f51fc1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc1396e0 con 0x7f51fc1430c0 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc138530 con 0x7f51fc0a98f0 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc1399c0 con 0x7f51fc13f710 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:38.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:38.989+0000 7f520470e640 10 monclient(hunting): authenticate will time out at 7598.211739s 2024-03-25T18:26:39.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.001+0000 7f5201ca9640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f51fc13f710 0x7f51fc1da850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:51556/0 (socket says 172.21.15.22:51556) 2024-03-25T18:26:39.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.001+0000 7f5201ca9640 1 -- 172.21.15.22:0/2906670558 learned_addr learned my addr 172.21.15.22:0/2906670558 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:39.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.002+0000 7f51f2ffd640 10 client.?.objecter ms_handle_connect 0x7f51fc13f710 2024-03-25T18:26:39.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.002+0000 7f51f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:39.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.007+0000 7f51f2ffd640 10 client.?.objecter ms_handle_connect 0x7f51fc0a98f0 2024-03-25T18:26:39.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.007+0000 7f51f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:39.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.022+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806303105 0 0) 0x7f51fc1399c0 con 0x7f51fc13f710 2024-03-25T18:26:39.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.022+0000 7f51f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.022+0000 7f51f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.022+0000 7f51f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.022+0000 7f51f2ffd640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:26:39.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.022+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e0004eb0 con 0x7f51fc13f710 2024-03-25T18:26:39.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.025+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958243983 0 0) 0x7f51fc138530 con 0x7f51fc0a98f0 2024-03-25T18:26:39.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.025+0000 7f51f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.025+0000 7f51f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.025+0000 7f51f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.025+0000 7f51f2ffd640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:26:39.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.025+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51fc1399c0 con 0x7f51fc0a98f0 2024-03-25T18:26:39.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.029+0000 7f51f2ffd640 10 client.?.objecter ms_handle_connect 0x7f51fc1430c0 2024-03-25T18:26:39.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.029+0000 7f51f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:39.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.045+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616469107 0 0) 0x7f51fc1399c0 con 0x7f51fc0a98f0 2024-03-25T18:26:39.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.045+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51fc138530 con 0x7f51fc0a98f0 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.045+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f51f8001930 con 0x7f51fc0a98f0 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.045+0000 7f51f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.045+0000 7f51f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.045+0000 7f51f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:39.046 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:39.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.045+0000 7f51f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:39.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.046+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331056828 0 0) 0x7f51fc1396e0 con 0x7f51fc1430c0 2024-03-25T18:26:39.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.046+0000 7f51f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:39.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.046+0000 7f51f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:39.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.046+0000 7f51f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:39.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.046+0000 7f51f2ffd640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:26:39.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.046+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51fc1399c0 con 0x7f51fc1430c0 2024-03-25T18:26:39.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.054+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865327378 0 0) 0x7f51e0004eb0 con 0x7f51fc13f710 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.054+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51fc1396e0 con 0x7f51fc13f710 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.054+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f51ec0019d0 con 0x7f51fc13f710 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.054+0000 7f51f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.054+0000 7f51f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.054+0000 7f51f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:39.055 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4141567341 0 0) 0x7f51fc138530 con 0x7f51fc0a98f0 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 >> v1:172.21.15.22:6790/0 conn(0x7f51fc13f710 legacy=0x7f51fc1da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 >> v1:172.21.15.22:6789/0 conn(0x7f51fc1430c0 legacy=0x7f51fc1e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 1 monclient: found mon.b 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc138a50 con 0x7f51fc0a98f0 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f51f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f520470e640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:26:39.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f520470e640 10 monclient: _renew_subs 2024-03-25T18:26:39.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f520470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:39.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.055+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc1380a0 con 0x7f51fc0a98f0 2024-03-25T18:26:39.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.056+0000 7f520470e640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:39.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.056+0000 7f520470e640 10 monclient: _renew_subs 2024-03-25T18:26:39.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.056+0000 7f520470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:39.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.056+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc1db140 con 0x7f51fc0a98f0 2024-03-25T18:26:39.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.056+0000 7f520470e640 1 librados: init done 2024-03-25T18:26:39.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.057+0000 7f520470e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:39.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.057+0000 7f520470e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:39.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.057+0000 7f520470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:39.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.057+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51fc138ca0 con 0x7f51fc0a98f0 2024-03-25T18:26:39.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.349+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51f8003610 con 0x7f51fc0a98f0 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.349+0000 7f51f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.349+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f51f80037c0 con 0x7f51fc0a98f0 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.349+0000 7f51f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.349+0000 7f51f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.349+0000 7f51f2ffd640 10 monclient: dump: 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:39.350 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:39.351 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:39.351 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:39.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.349+0000 7f51f3fff640 4 set_mon_vals no callback set 2024-03-25T18:26:39.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.350+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f51f80039e0 con 0x7f51fc0a98f0 2024-03-25T18:26:39.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.350+0000 7f5201ca9640 10 monclient: get_auth_request con 0x7f51e006dfb0 auth_method 0 2024-03-25T18:26:39.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.351+0000 7f51f2ffd640 10 client.4558.objecter ms_handle_connect 0x7f51e006dfb0 2024-03-25T18:26:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f51f8053440 con 0x7f51fc0a98f0 2024-03-25T18:26:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 10 client.4558.objecter ms_dispatch 0x7f51fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T18:26:39.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51f804ef50 con 0x7f51fc0a98f0 2024-03-25T18:26:39.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:39.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.355+0000 7f51f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:39.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.610+0000 7f520470e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:39.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.610+0000 7f520470e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:39.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.610+0000 7f520470e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:39.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.610+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51fc0627b0 con 0x7f51fc0a98f0 2024-03-25T18:26:39.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.745+0000 7f51f2ffd640 1 -- 172.21.15.22:0/2906670558 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f51f8050e80 con 0x7f51fc0a98f0 2024-03-25T18:26:39.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.745+0000 7f51f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:39.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.745+0000 7f51f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:39.746 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:39.746 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:39.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 10 librados: watch_flush enter 2024-03-25T18:26:39.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 10 librados: watch_flush exit 2024-03-25T18:26:39.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 10 client.4558.objecter successfully canceled tick 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 >> v1:172.21.15.33:6800/215350958 conn(0x7f51e006dfb0 legacy=0x7f51e0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 10 monclient: shutdown 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.747+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 >> v1:172.21.15.33:6789/0 conn(0x7f51fc0a98f0 legacy=0x7f51fc137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.748+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 shutdown_connections 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.748+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 >> 172.21.15.22:0/2906670558 conn(0x7f51fc0b7ea0 msgr2=0x7f51fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.748+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 shutdown_connections 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.748+0000 7f520470e640 1 -- 172.21.15.22:0/2906670558 wait complete. 2024-03-25T18:26:39.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:39.748+0000 7f520470e640 1 librados: shutdown 2024-03-25T18:26:39.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:46.568 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.682+0000 7fb302df6640 10 monclient: get_monmap_and_config 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.682+0000 7fb302df6640 10 monclient: build_initial_monmap 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.682+0000 7fb302df6640 10 monclient: monmap: 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:46.682776+0000 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:46.682776+0000 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:46.683 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:46.684 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:46.684 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.683+0000 7fb302df6640 1 Processor -- start 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.683+0000 7fb302df6640 1 -- start start 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.683+0000 7fb302df6640 10 monclient: init 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc0705b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc1402d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 10 monclient(hunting): picked mon.noname-b con 0x7fb2fc149ae0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0718c0 con 0x7fb2fc1402d0 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc0714c0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc072b50 con 0x7fb2fc149ae0 2024-03-25T18:26:46.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:46.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.684+0000 7fb302df6640 10 monclient(hunting): authenticate will time out at 7605.907201s 2024-03-25T18:26:46.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.695+0000 7fb2fbfff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb2fc1402d0 0x7fb2fc140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55866/0 (socket says 172.21.15.22:55866) 2024-03-25T18:26:46.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.695+0000 7fb2fbfff640 1 -- 172.21.15.22:0/1427349515 learned_addr learned my addr 172.21.15.22:0/1427349515 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.717+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713718130 0 0) 0x7fb2fc0718c0 con 0x7fb2fc1402d0 2024-03-25T18:26:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.717+0000 7fb2fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.717+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.717+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.717+0000 7fb2fb7fe640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:26:46.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.717+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ec003520 con 0x7fb2fc1402d0 2024-03-25T18:26:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.722+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886691932 0 0) 0x7fb2fc0714c0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.722+0000 7fb2fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.722+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.722+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.722+0000 7fb2fb7fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:26:46.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.722+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc0718c0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.731+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762153389 0 0) 0x7fb2fc072b50 con 0x7fb2fc149ae0 2024-03-25T18:26:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.731+0000 7fb2fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.731+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.731+0000 7fb2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.731+0000 7fb2fb7fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:26:46.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.731+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc0714c0 con 0x7fb2fc149ae0 2024-03-25T18:26:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2532154319 0 0) 0x7fb2fc0718c0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 >> v1:172.21.15.22:6790/0 conn(0x7fb2fc149ae0 legacy=0x7fb2fc14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 >> v1:172.21.15.22:6789/0 conn(0x7fb2fc1402d0 legacy=0x7fb2fc140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc13dd10 con 0x7fb2fc0705b0 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb2f0001670 con 0x7fb2fc0705b0 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb2fb7fe640 10 monclient: dump: 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.739 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb302df6640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:26:46.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.738+0000 7fb302df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:46.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.767+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2f00017d0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.767+0000 7fb2fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:46.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.767+0000 7fb2fb7fe640 1 -- 172.21.15.22:0/1427349515 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb2f0001be0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.767+0000 7fb2fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.767+0000 7fb2fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:46.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.767+0000 7fb2fb7fe640 10 monclient: dump: 2024-03-25T18:26:46.768 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.768+0000 7fb302df6640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:46.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.768+0000 7fb302df6640 4 set_mon_vals no callback set 2024-03-25T18:26:46.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.768+0000 7fb302df6640 10 monclient: shutdown 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.768+0000 7fb302df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.768+0000 7fb302df6640 1 -- 172.21.15.22:0/1427349515 >> v1:172.21.15.33:6789/0 conn(0x7fb2fc0705b0 legacy=0x7fb2fc13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.768+0000 7fb302df6640 1 -- 172.21.15.22:0/1427349515 shutdown_connections 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.768+0000 7fb302df6640 1 -- 172.21.15.22:0/1427349515 >> 172.21.15.22:0/1427349515 conn(0x7fb2fc0b5d10 msgr2=0x7fb2fc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.769+0000 7fb302df6640 1 -- 172.21.15.22:0/1427349515 shutdown_connections 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.769+0000 7fb302df6640 1 -- 172.21.15.22:0/1427349515 wait complete. 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.769+0000 7fb302df6640 10 monclient: build_initial_monmap 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.769+0000 7fb302df6640 10 monclient: monmap: 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:46.772 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.769+0000 7fb302df6640 1 librados: starting msgr at 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.769+0000 7fb302df6640 1 librados: starting objecter 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 1 Processor -- start 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 1 -- start start 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 1 librados: setting wanted keys 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 1 librados: calling monclient init 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient: init 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:46.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient(hunting): picked mon.noname-c con 0x7fb2fc0705b0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient(hunting): picked mon.noname-a con 0x7fb2fc1402d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient(hunting): picked mon.noname-b con 0x7fb2fc149ae0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc142bc0 con 0x7fb2fc1402d0 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc134870 con 0x7fb2fc149ae0 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc143050 con 0x7fb2fc0705b0 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:46.774 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.770+0000 7fb302df6640 10 monclient(hunting): authenticate will time out at 7605.993453s 2024-03-25T18:26:46.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.784+0000 7fb301393640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fb2fc149ae0 0x7fb2fc134160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:56228/0 (socket says 172.21.15.22:56228) 2024-03-25T18:26:46.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.784+0000 7fb301393640 1 -- 172.21.15.22:0/1932055490 learned_addr learned my addr 172.21.15.22:0/1932055490 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:46.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.789+0000 7fb2f97fa640 10 client.?.objecter ms_handle_connect 0x7fb2fc149ae0 2024-03-25T18:26:46.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.789+0000 7fb2f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:46.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.789+0000 7fb2f97fa640 10 client.?.objecter ms_handle_connect 0x7fb2fc0705b0 2024-03-25T18:26:46.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.789+0000 7fb2f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:46.794 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.793+0000 7fb2f97fa640 10 client.?.objecter ms_handle_connect 0x7fb2fc1402d0 2024-03-25T18:26:46.794 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.793+0000 7fb2f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:46.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.795+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554603700 0 0) 0x7fb2fc134870 con 0x7fb2fc149ae0 2024-03-25T18:26:46.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.795+0000 7fb2f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.795+0000 7fb2f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.795+0000 7fb2f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.795+0000 7fb2f97fa640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:26:46.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.795+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e0004eb0 con 0x7fb2fc149ae0 2024-03-25T18:26:46.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.797+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330491718 0 0) 0x7fb2fc143050 con 0x7fb2fc0705b0 2024-03-25T18:26:46.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.797+0000 7fb2f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.797+0000 7fb2f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.797+0000 7fb2f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.797+0000 7fb2f97fa640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:26:46.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.797+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc134870 con 0x7fb2fc0705b0 2024-03-25T18:26:46.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.807+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230235195 0 0) 0x7fb2fc142bc0 con 0x7fb2fc1402d0 2024-03-25T18:26:46.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.807+0000 7fb2f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:26:46.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.807+0000 7fb2f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:46.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.807+0000 7fb2f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:46.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.807+0000 7fb2f97fa640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:26:46.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.807+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc143050 con 0x7fb2fc1402d0 2024-03-25T18:26:46.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.813+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86984287 0 0) 0x7fb2fc134870 con 0x7fb2fc0705b0 2024-03-25T18:26:46.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.813+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2fc142bc0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.814+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb2f0001ae0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.814+0000 7fb2f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.814+0000 7fb2f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:46.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.814+0000 7fb2f97fa640 10 monclient(hunting): dump: 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:46.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.814+0000 7fb2f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.816+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3858632877 0 0) 0x7fb2e0004eb0 con 0x7fb2fc149ae0 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.816+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2fc134870 con 0x7fb2fc149ae0 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.816+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb2f4001c30 con 0x7fb2fc149ae0 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.816+0000 7fb2f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.816+0000 7fb2f97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.816+0000 7fb2f97fa640 10 monclient(hunting): dump: 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:46.817 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.818 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:46.818 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:46.818 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:46.818 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:46.818 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:46.818 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:46.818 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 240789319 0 0) 0x7fb2fc142bc0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 >> v1:172.21.15.33:6789/0 conn(0x7fb2fc149ae0 legacy=0x7fb2fc134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 >> v1:172.21.15.22:6789/0 conn(0x7fb2fc1402d0 legacy=0x7fb2fc133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 1 monclient: found mon.c 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc143e10 con 0x7fb2fc0705b0 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 10 monclient: _finish_auth 0 2024-03-25T18:26:46.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb2f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 10 monclient: _renew_subs 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc06d7a0 con 0x7fb2fc0705b0 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 10 monclient: _renew_subs 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc142350 con 0x7fb2fc0705b0 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.819+0000 7fb302df6640 1 librados: init done 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.820+0000 7fb302df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.820+0000 7fb302df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.820+0000 7fb302df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:26:46.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:46.820+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc06dab0 con 0x7fb2fc0705b0 2024-03-25T18:26:47.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.136+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2f0001210 con 0x7fb2fc0705b0 2024-03-25T18:26:47.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.136+0000 7fb2f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:47.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.136+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb2f0003a40 con 0x7fb2fc0705b0 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.136+0000 7fb2f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.136+0000 7fb2f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.136+0000 7fb2f97fa640 10 monclient: dump: 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.136+0000 7fb2faffd640 4 set_mon_vals no callback set 2024-03-25T18:26:47.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.137+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fb2f0019a10 con 0x7fb2fc0705b0 2024-03-25T18:26:47.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.138+0000 7fb2fbfff640 10 monclient: get_auth_request con 0x7fb2e006df20 auth_method 0 2024-03-25T18:26:47.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.138+0000 7fb2f97fa640 10 client.4568.objecter ms_handle_connect 0x7fb2e006df20 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fb2f00536a0 con 0x7fb2fc0705b0 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 10 client.4568.objecter ms_dispatch 0x7fb2fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2f004efe0 con 0x7fb2fc0705b0 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:47.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.146+0000 7fb2f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:47.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.364+0000 7fb302df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:47.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.364+0000 7fb302df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:47.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.364+0000 7fb302df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:26:47.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.364+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2fc0627b0 con 0x7fb2fc0705b0 2024-03-25T18:26:47.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.392+0000 7fb2f97fa640 1 -- 172.21.15.22:0/1932055490 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fb2f00510e0 con 0x7fb2fc0705b0 2024-03-25T18:26:47.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.392+0000 7fb2f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:47.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.392+0000 7fb2f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:47.394 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:47.394 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:47.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 10 librados: watch_flush enter 2024-03-25T18:26:47.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 10 librados: watch_flush exit 2024-03-25T18:26:47.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 10 client.4568.objecter successfully canceled tick 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 >> v1:172.21.15.33:6800/215350958 conn(0x7fb2e006df20 legacy=0x7fb2e00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 10 monclient: shutdown 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 >> v1:172.21.15.22:6790/0 conn(0x7fb2fc0705b0 legacy=0x7fb2fc1daa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 shutdown_connections 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.394+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 >> 172.21.15.22:0/1932055490 conn(0x7fb2fc0b5d10 msgr2=0x7fb2fc13c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.395+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 shutdown_connections 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.395+0000 7fb302df6640 1 -- 172.21.15.22:0/1932055490 wait complete. 2024-03-25T18:26:47.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:47.395+0000 7fb302df6640 1 librados: shutdown 2024-03-25T18:26:47.409 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:54.215 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:54.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.331+0000 7f973a1a3640 10 monclient: get_monmap_and_config 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.331+0000 7f973a1a3640 10 monclient: build_initial_monmap 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.331+0000 7f973a1a3640 10 monclient: monmap: 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:26:54.332513+0000 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:26:54.332513+0000 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:54.333 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:26:54.334 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:26:54.334 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:54.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.332+0000 7f973a1a3640 1 Processor -- start 2024-03-25T18:26:54.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.332+0000 7f973a1a3640 1 -- start start 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.332+0000 7f973a1a3640 10 monclient: init 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 10 monclient(hunting): picked mon.noname-b con 0x7f97340a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 10 monclient(hunting): picked mon.noname-a con 0x7f973413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f97341430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9734142520 con 0x7f973413f710 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340b28a0 con 0x7f97341430c0 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97340b2a30 con 0x7f97340a98f0 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:54.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.333+0000 7f973a1a3640 10 monclient(hunting): authenticate will time out at 7613.556843s 2024-03-25T18:26:54.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.339+0000 7f9733fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f97341430c0 0x7f973414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:49950/0 (socket says 172.21.15.22:49950) 2024-03-25T18:26:54.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.339+0000 7f9733fff640 1 -- 172.21.15.22:0/2960026963 learned_addr learned my addr 172.21.15.22:0/2960026963 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:54.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.354+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251508894 0 0) 0x7f97340b28a0 con 0x7f97341430c0 2024-03-25T18:26:54.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.354+0000 7f97327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:54.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.354+0000 7f97327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:54.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.354+0000 7f97327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:54.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.354+0000 7f97327fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:26:54.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.354+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f971c003520 con 0x7f97341430c0 2024-03-25T18:26:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.355+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522165437 0 0) 0x7f97340b2a30 con 0x7f97340a98f0 2024-03-25T18:26:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.355+0000 7f97327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.355+0000 7f97327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.355+0000 7f97327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.355+0000 7f97327fc640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:26:54.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.355+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97340b28a0 con 0x7f97340a98f0 2024-03-25T18:26:54.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.357+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250511423 0 0) 0x7f9734142520 con 0x7f973413f710 2024-03-25T18:26:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.357+0000 7f97327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.357+0000 7f97327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.357+0000 7f97327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.357+0000 7f97327fc640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:26:54.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.357+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97340b2a30 con 0x7f973413f710 2024-03-25T18:26:54.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.360+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1096158565 0 0) 0x7f971c003520 con 0x7f97341430c0 2024-03-25T18:26:54.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.360+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 >> v1:172.21.15.22:6790/0 conn(0x7f97340a98f0 legacy=0x7f97340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.360+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 >> v1:172.21.15.22:6789/0 conn(0x7f973413f710 legacy=0x7f97341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:54.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 1 monclient: found mon.noname-c 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340b44c0 con 0x7f97341430c0 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f972c001a10 con 0x7f97341430c0 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f973a1a3640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f973a1a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f97327fc640 10 monclient: dump: 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:54.363 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:54.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.361+0000 7f973a1a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:54.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f972c001b70 con 0x7f97341430c0 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f97327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f97327fc640 1 -- 172.21.15.22:0/2960026963 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f972c001f80 con 0x7f97341430c0 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f973a1a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f973a1a3640 4 set_mon_vals no callback set 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f97327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f97327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.369+0000 7f97327fc640 10 monclient: dump: 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.371 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:54.372 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:54.372 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:54.372 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:54.372 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:54.372 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.370+0000 7f973a1a3640 10 monclient: shutdown 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.370+0000 7f973a1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.370+0000 7f973a1a3640 1 -- 172.21.15.22:0/2960026963 >> v1:172.21.15.33:6789/0 conn(0x7f97341430c0 legacy=0x7f973414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.370+0000 7f973a1a3640 1 -- 172.21.15.22:0/2960026963 shutdown_connections 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.370+0000 7f973a1a3640 1 -- 172.21.15.22:0/2960026963 >> 172.21.15.22:0/2960026963 conn(0x7f97340b7ea0 msgr2=0x7f97340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.370+0000 7f973a1a3640 1 -- 172.21.15.22:0/2960026963 shutdown_connections 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.370+0000 7f973a1a3640 1 -- 172.21.15.22:0/2960026963 wait complete. 2024-03-25T18:26:54.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.371+0000 7f973a1a3640 10 monclient: build_initial_monmap 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.371+0000 7f973a1a3640 10 monclient: monmap: 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.371+0000 7f973a1a3640 1 librados: starting msgr at 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.371+0000 7f973a1a3640 1 librados: starting objecter 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.371+0000 7f973a1a3640 1 Processor -- start 2024-03-25T18:26:54.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 1 -- start start 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 1 librados: setting wanted keys 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 1 librados: calling monclient init 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient: init 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient(hunting): picked mon.noname-a con 0x7f97340a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient(hunting): picked mon.noname-b con 0x7f973413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f97341430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97341db170 con 0x7f97340a98f0 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97341dac50 con 0x7f973413f710 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97341dade0 con 0x7f97341430c0 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:54.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.372+0000 7f973a1a3640 10 monclient(hunting): authenticate will time out at 7613.596526s 2024-03-25T18:26:54.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.377+0000 7f9732ffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f973413f710 0x7f97341d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:49998/0 (socket says 172.21.15.22:49998) 2024-03-25T18:26:54.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.377+0000 7f9732ffd640 1 -- 172.21.15.22:0/3805216805 learned_addr learned my addr 172.21.15.22:0/3805216805 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:26:54.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.378+0000 7f971bfff640 10 client.?.objecter ms_handle_connect 0x7f973413f710 2024-03-25T18:26:54.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.378+0000 7f971bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:54.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.383+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389271713 0 0) 0x7f97341dac50 con 0x7f973413f710 2024-03-25T18:26:54.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.383+0000 7f971bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:54.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.383+0000 7f971bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:54.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.383+0000 7f971bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:54.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.383+0000 7f971bfff640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:26:54.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.383+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9720004eb0 con 0x7f973413f710 2024-03-25T18:26:54.394 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.393+0000 7f971bfff640 10 client.?.objecter ms_handle_connect 0x7f97341430c0 2024-03-25T18:26:54.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.393+0000 7f971bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:54.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.395+0000 7f971bfff640 10 client.?.objecter ms_handle_connect 0x7f97340a98f0 2024-03-25T18:26:54.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.395+0000 7f971bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:54.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.403+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446947491 0 0) 0x7f97341dade0 con 0x7f97341430c0 2024-03-25T18:26:54.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.403+0000 7f971bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:54.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.403+0000 7f971bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:54.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.403+0000 7f971bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:54.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.403+0000 7f971bfff640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:26:54.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.403+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97341dac50 con 0x7f97341430c0 2024-03-25T18:26:54.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.404+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4287247840 0 0) 0x7f9720004eb0 con 0x7f973413f710 2024-03-25T18:26:54.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.404+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97341dade0 con 0x7f973413f710 2024-03-25T18:26:54.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.404+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9728001a10 con 0x7f973413f710 2024-03-25T18:26:54.405 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.404+0000 7f971bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.404+0000 7f971bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.404+0000 7f971bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:54.406 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.404+0000 7f971bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:54.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.406+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217931590 0 0) 0x7f97341db170 con 0x7f97340a98f0 2024-03-25T18:26:54.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.406+0000 7f971bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:54.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.406+0000 7f971bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:54.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.406+0000 7f971bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:54.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.406+0000 7f971bfff640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:26:54.408 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.406+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9720004eb0 con 0x7f97340a98f0 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.425+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2125830930 0 0) 0x7f9720004eb0 con 0x7f97340a98f0 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.425+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97341db170 con 0x7f97340a98f0 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.425+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9724001930 con 0x7f97340a98f0 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.425+0000 7f971bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.425+0000 7f971bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.425+0000 7f971bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:54.427 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:54.428 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:54.428 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:54.428 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:54.428 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:54.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.430+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 698548621 0 0) 0x7f97341dade0 con 0x7f973413f710 2024-03-25T18:26:54.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 >> v1:172.21.15.22:6790/0 conn(0x7f97341430c0 legacy=0x7f9734132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 >> v1:172.21.15.22:6789/0 conn(0x7f97340a98f0 legacy=0x7f97341394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 1 monclient: found mon.b 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f973413c170 con 0x7f973413f710 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f971bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 10 monclient: _renew_subs 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f973413c420 con 0x7f973413f710 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 10 monclient: _renew_subs 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9734133860 con 0x7f973413f710 2024-03-25T18:26:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.431+0000 7f973a1a3640 1 librados: init done 2024-03-25T18:26:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.432+0000 7f973a1a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.432+0000 7f973a1a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.432+0000 7f973a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.432+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97341daf70 con 0x7f973413f710 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.721+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9728001c70 con 0x7f973413f710 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.721+0000 7f971bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.721+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f9728003a60 con 0x7f973413f710 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.721+0000 7f971bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.721+0000 7f971bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.721+0000 7f971bfff640 10 monclient: dump: 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.723 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.721+0000 7f9731ffb640 4 set_mon_vals no callback set 2024-03-25T18:26:54.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.722+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f9728003c40 con 0x7f973413f710 2024-03-25T18:26:54.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.723+0000 7f97337fe640 10 monclient: get_auth_request con 0x7f972006dfb0 auth_method 0 2024-03-25T18:26:54.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.723+0000 7f971bfff640 10 client.4570.objecter ms_handle_connect 0x7f972006dfb0 2024-03-25T18:26:54.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f9728054890 con 0x7f973413f710 2024-03-25T18:26:54.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 10 client.4570.objecter ms_dispatch 0x7f9734000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:54.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:54.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:54.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:26:54.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97280501d0 con 0x7f973413f710 2024-03-25T18:26:54.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:54.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.728+0000 7f971bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:54.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.974+0000 7f973a1a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.974+0000 7f973a1a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:54.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.974+0000 7f973a1a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:26:54.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:54.974+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97340627b0 con 0x7f973413f710 2024-03-25T18:26:55.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.139+0000 7f971bfff640 1 -- 172.21.15.22:0/3805216805 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f97280522d0 con 0x7f973413f710 2024-03-25T18:26:55.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.139+0000 7f971bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:55.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.139+0000 7f971bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:55.141 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:26:55.141 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 10 librados: watch_flush enter 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 10 librados: watch_flush exit 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 >> v1:172.21.15.33:6800/215350958 conn(0x7f972006dfb0 legacy=0x7f9720070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 10 monclient: shutdown 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 >> v1:172.21.15.33:6789/0 conn(0x7f973413f710 legacy=0x7f97341d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:55.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 shutdown_connections 2024-03-25T18:26:55.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 >> 172.21.15.22:0/3805216805 conn(0x7f97340b7ea0 msgr2=0x7f973414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:55.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.141+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 shutdown_connections 2024-03-25T18:26:55.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.142+0000 7f973a1a3640 1 -- 172.21.15.22:0/3805216805 wait complete. 2024-03-25T18:26:55.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:26:55.142+0000 7f973a1a3640 1 librados: shutdown 2024-03-25T18:26:55.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:58.690 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:01.351 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.690 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:01.308 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.693 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:01.355 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.728 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:01.389 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.750 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:01.368 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.751 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:01.413 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.765 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:01.427 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.775 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:01.393 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:58.811 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:01.473 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:01.962 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:02.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.076+0000 7f5a8530f640 10 monclient: get_monmap_and_config 2024-03-25T18:27:02.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.076+0000 7f5a8530f640 10 monclient: build_initial_monmap 2024-03-25T18:27:02.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.076+0000 7f5a8530f640 10 monclient: monmap: 2024-03-25T18:27:02.078 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:02.078 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:02.078088+0000 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:02.078088+0000 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:02.079 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.077+0000 7f5a8530f640 1 Processor -- start 2024-03-25T18:27:02.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.077+0000 7f5a8530f640 1 -- start start 2024-03-25T18:27:02.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.077+0000 7f5a8530f640 10 monclient: init 2024-03-25T18:27:02.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 10 monclient(hunting): picked mon.noname-b con 0x7f5a80061830 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 10 monclient(hunting): picked mon.noname-a con 0x7f5a800ae840 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8013fbd0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a800ae1c0 con 0x7f5a800ae840 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8006f7d0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8006fa40 con 0x7f5a80061830 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:02.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.078+0000 7f5a8530f640 10 monclient(hunting): authenticate will time out at 7621.302033s 2024-03-25T18:27:02.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.081+0000 7f5a7f7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f5a8013fbd0 0x7f5a80145f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:54682/0 (socket says 172.21.15.22:54682) 2024-03-25T18:27:02.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.081+0000 7f5a7f7fe640 1 -- 172.21.15.22:0/774082204 learned_addr learned my addr 172.21.15.22:0/774082204 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:02.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.088+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268991694 0 0) 0x7f5a8006f7d0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.088+0000 7f5a7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.088+0000 7f5a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.088+0000 7f5a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.088+0000 7f5a7e7fc640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:27:02.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.088+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c003520 con 0x7f5a8013fbd0 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196598545 0 0) 0x7f5a6c003520 con 0x7f5a8013fbd0 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 >> v1:172.21.15.22:6790/0 conn(0x7f5a80061830 legacy=0x7f5a8013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 >> v1:172.21.15.22:6789/0 conn(0x7f5a800ae840 legacy=0x7f5a80145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:27:02.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a80073040 con 0x7f5a8013fbd0 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5a78001a10 con 0x7f5a8013fbd0 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a8530f640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a8530f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a7e7fc640 10 monclient: dump: 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.103 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:02.104 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:02.104 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:02.104 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:02.104 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.100+0000 7f5a8530f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a78001b70 con 0x7f5a8013fbd0 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a7e7fc640 1 -- 172.21.15.22:0/774082204 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5a78001f00 con 0x7f5a8013fbd0 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a8530f640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a8530f640 4 set_mon_vals no callback set 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a7e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:02.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.109+0000 7f5a7e7fc640 10 monclient: dump: 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.110+0000 7f5a8530f640 10 monclient: shutdown 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.110+0000 7f5a8530f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:02.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.110+0000 7f5a8530f640 1 -- 172.21.15.22:0/774082204 >> v1:172.21.15.33:6789/0 conn(0x7f5a8013fbd0 legacy=0x7f5a80145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.110+0000 7f5a8530f640 1 -- 172.21.15.22:0/774082204 shutdown_connections 2024-03-25T18:27:02.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.110+0000 7f5a8530f640 1 -- 172.21.15.22:0/774082204 >> 172.21.15.22:0/774082204 conn(0x7f5a800b0ef0 msgr2=0x7f5a800b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:02.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.110+0000 7f5a8530f640 1 -- 172.21.15.22:0/774082204 shutdown_connections 2024-03-25T18:27:02.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.110+0000 7f5a8530f640 1 -- 172.21.15.22:0/774082204 wait complete. 2024-03-25T18:27:02.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.111+0000 7f5a8530f640 10 monclient: build_initial_monmap 2024-03-25T18:27:02.115 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.111+0000 7f5a8530f640 10 monclient: monmap: 2024-03-25T18:27:02.115 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:02.115 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:02.115 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.116 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.111+0000 7f5a8530f640 1 librados: starting msgr at 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.111+0000 7f5a8530f640 1 librados: starting objecter 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.111+0000 7f5a8530f640 1 Processor -- start 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 1 -- start start 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 1 librados: setting wanted keys 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 1 librados: calling monclient init 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient: init 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient(hunting): picked mon.noname-b con 0x7f5a80061830 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient(hunting): picked mon.noname-c con 0x7f5a800ae840 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8013fbd0 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:02.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8013bf40 con 0x7f5a8013fbd0 2024-03-25T18:27:02.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a801d3ec0 con 0x7f5a80061830 2024-03-25T18:27:02.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a801d4050 con 0x7f5a800ae840 2024-03-25T18:27:02.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:02.118 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.112+0000 7f5a8530f640 10 monclient(hunting): authenticate will time out at 7621.336414s 2024-03-25T18:27:02.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.121+0000 7f5a77fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5a800ae840 0x7f5a801d7860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:32962/0 (socket says 172.21.15.22:32962) 2024-03-25T18:27:02.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.121+0000 7f5a77fff640 1 -- 172.21.15.22:0/4072191388 learned_addr learned my addr 172.21.15.22:0/4072191388 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:02.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.122+0000 7f5a777fe640 10 client.?.objecter ms_handle_connect 0x7f5a800ae840 2024-03-25T18:27:02.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.122+0000 7f5a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:02.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.124+0000 7f5a777fe640 10 client.?.objecter ms_handle_connect 0x7f5a80061830 2024-03-25T18:27:02.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.124+0000 7f5a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:02.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.126+0000 7f5a777fe640 10 client.?.objecter ms_handle_connect 0x7f5a8013fbd0 2024-03-25T18:27:02.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.126+0000 7f5a777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:02.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.133+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651716143 0 0) 0x7f5a8013bf40 con 0x7f5a8013fbd0 2024-03-25T18:27:02.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.133+0000 7f5a777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.133+0000 7f5a777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.133+0000 7f5a777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.133+0000 7f5a777fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:27:02.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.133+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a60004eb0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.140+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3177836898 0 0) 0x7f5a800ae1c0 con 0x7f5a800ae840 2024-03-25T18:27:02.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.140+0000 7f5a777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.140+0000 7f5a777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.140+0000 7f5a777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.140+0000 7f5a777fe640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:27:02.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.140+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8013bf40 con 0x7f5a800ae840 2024-03-25T18:27:02.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.145+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194105739 0 0) 0x7f5a801d3ec0 con 0x7f5a80061830 2024-03-25T18:27:02.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.145+0000 7f5a777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:02.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.145+0000 7f5a777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:02.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.145+0000 7f5a777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:02.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.145+0000 7f5a777fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:27:02.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.145+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a800ae1c0 con 0x7f5a80061830 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.157+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612734487 0 0) 0x7f5a60004eb0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.157+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a801d3ec0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.157+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5a78001e80 con 0x7f5a8013fbd0 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.157+0000 7f5a777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.157+0000 7f5a777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.157+0000 7f5a777fe640 10 monclient(hunting): dump: 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:02.159 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:02.160 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:02.160 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:02.160 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.157+0000 7f5a777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:02.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.165+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 553208961 0 0) 0x7f5a800ae1c0 con 0x7f5a80061830 2024-03-25T18:27:02.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.165+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a60004eb0 con 0x7f5a80061830 2024-03-25T18:27:02.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.165+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5a70001790 con 0x7f5a80061830 2024-03-25T18:27:02.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.165+0000 7f5a777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.165+0000 7f5a777fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:02.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.165+0000 7f5a777fe640 10 monclient(hunting): dump: 2024-03-25T18:27:02.168 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:02.168 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:02.168 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.168 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.169 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.169 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:02.169 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:02.169 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:02.169 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:02.169 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.167+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3284551830 0 0) 0x7f5a8013bf40 con 0x7f5a800ae840 2024-03-25T18:27:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.167+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a800ae1c0 con 0x7f5a800ae840 2024-03-25T18:27:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.167+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5a680017e0 con 0x7f5a800ae840 2024-03-25T18:27:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.167+0000 7f5a777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.167+0000 7f5a777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:02.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.167+0000 7f5a777fe640 10 monclient(hunting): dump: 2024-03-25T18:27:02.171 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:02.171 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:02.172 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.172 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.172 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.172 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:02.172 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:02.172 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:02.173 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:02.173 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2037951821 0 0) 0x7f5a801d3ec0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 >> v1:172.21.15.22:6790/0 conn(0x7f5a800ae840 legacy=0x7f5a801d7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 >> v1:172.21.15.33:6789/0 conn(0x7f5a80061830 legacy=0x7f5a8013a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:02.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 1 monclient: found mon.a 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8013ad90 con 0x7f5a8013fbd0 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 5 monclient: authenticate success, global_id 4566 2024-03-25T18:27:02.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 10 monclient: _renew_subs 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a801d7f70 con 0x7f5a8013fbd0 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 10 monclient: _renew_subs 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8013a130 con 0x7f5a8013fbd0 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.178+0000 7f5a8530f640 1 librados: init done 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.180+0000 7f5a8530f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.180+0000 7f5a8530f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.180+0000 7f5a8530f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:02.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.180+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a801d7580 con 0x7f5a8013fbd0 2024-03-25T18:27:02.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a78001420 con 0x7f5a8013fbd0 2024-03-25T18:27:02.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:02.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5a78004e10 con 0x7f5a8013fbd0 2024-03-25T18:27:02.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a777fe640 10 monclient: dump: 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:02.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a7dffb640 4 set_mon_vals no callback set 2024-03-25T18:27:02.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.506+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f5a7801ade0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.507+0000 7f5a7effd640 10 monclient: get_auth_request con 0x7f5a6006e0b0 auth_method 0 2024-03-25T18:27:02.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.507+0000 7f5a777fe640 10 client.4566.objecter ms_handle_connect 0x7f5a6006e0b0 2024-03-25T18:27:02.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.513+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f5a78054980 con 0x7f5a8013fbd0 2024-03-25T18:27:02.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.513+0000 7f5a777fe640 10 client.4566.objecter ms_dispatch 0x7f5a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:02.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.513+0000 7f5a777fe640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:02.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.513+0000 7f5a777fe640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:02.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.513+0000 7f5a777fe640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-25T18:27:02.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.513+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a780502c0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.514+0000 7f5a777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:02.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.514+0000 7f5a777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:02.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.770+0000 7f5a8530f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:02.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.770+0000 7f5a8530f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:02.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.770+0000 7f5a8530f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:02.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.770+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a8013bd80 con 0x7f5a8013fbd0 2024-03-25T18:27:02.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.805+0000 7f5a777fe640 1 -- 172.21.15.22:0/4072191388 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f5a780523c0 con 0x7f5a8013fbd0 2024-03-25T18:27:02.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.805+0000 7f5a777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:02.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.805+0000 7f5a777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:02.807 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:02.807 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:02.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.807+0000 7f5a8530f640 10 librados: watch_flush enter 2024-03-25T18:27:02.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.807+0000 7f5a8530f640 10 librados: watch_flush exit 2024-03-25T18:27:02.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.807+0000 7f5a8530f640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.807+0000 7f5a8530f640 10 client.4566.objecter successfully canceled tick 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 >> v1:172.21.15.33:6800/215350958 conn(0x7f5a6006e0b0 legacy=0x7f5a60070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 10 monclient: shutdown 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 >> v1:172.21.15.22:6789/0 conn(0x7f5a8013fbd0 legacy=0x7f5a801d4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 shutdown_connections 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 >> 172.21.15.22:0/4072191388 conn(0x7f5a800b0ef0 msgr2=0x7f5a8013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 shutdown_connections 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 1 -- 172.21.15.22:0/4072191388 wait complete. 2024-03-25T18:27:02.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:02.808+0000 7f5a8530f640 1 librados: shutdown 2024-03-25T18:27:02.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:09.627 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.739+0000 7f53f0674640 10 monclient: get_monmap_and_config 2024-03-25T18:27:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.739+0000 7f53f0674640 10 monclient: build_initial_monmap 2024-03-25T18:27:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.739+0000 7f53f0674640 10 monclient: monmap: 2024-03-25T18:27:09.741 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:09.741 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:09.741 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:09.741249+0000 2024-03-25T18:27:09.741 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:09.741249+0000 2024-03-25T18:27:09.742 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:09.742 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:09.742 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:09.742 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:09.742 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:09.742 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.740+0000 7f53f0674640 1 Processor -- start 2024-03-25T18:27:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.740+0000 7f53f0674640 1 -- start start 2024-03-25T18:27:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.740+0000 7f53f0674640 10 monclient: init 2024-03-25T18:27:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 10 monclient(hunting): picked mon.noname-b con 0x7f53e8061830 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 10 monclient(hunting): picked mon.noname-c con 0x7f53e806e740 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 10 monclient(hunting): picked mon.noname-a con 0x7f53e80ae280 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e80ae830 con 0x7f53e80ae280 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e806f720 con 0x7f53e806e740 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e80ada80 con 0x7f53e8061830 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.741+0000 7f53f0674640 10 monclient(hunting): authenticate will time out at 7628.965401s 2024-03-25T18:27:09.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.747+0000 7f53edc0f640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f53e806e740 0x7f53e8145820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:47300/0 (socket says 172.21.15.22:47300) 2024-03-25T18:27:09.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.747+0000 7f53edc0f640 1 -- 172.21.15.22:0/822244271 learned_addr learned my addr 172.21.15.22:0/822244271 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:09.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.762+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857985970 0 0) 0x7f53e80ae830 con 0x7f53e80ae280 2024-03-25T18:27:09.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.762+0000 7f53ed40e640 10 monclient(hunting): _negotiate 2024-03-25T18:27:09.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.762+0000 7f53ed40e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:09.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.762+0000 7f53ed40e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:09.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.762+0000 7f53ed40e640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:27:09.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.762+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d4003520 con 0x7f53e80ae280 2024-03-25T18:27:09.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.772+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586656449 0 0) 0x7f53e80ada80 con 0x7f53e8061830 2024-03-25T18:27:09.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.772+0000 7f53ed40e640 10 monclient(hunting): _negotiate 2024-03-25T18:27:09.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.772+0000 7f53ed40e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:09.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.772+0000 7f53ed40e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:09.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.772+0000 7f53ed40e640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:27:09.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.772+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e80ae830 con 0x7f53e8061830 2024-03-25T18:27:09.778 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.777+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997239251 0 0) 0x7f53e806f720 con 0x7f53e806e740 2024-03-25T18:27:09.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.777+0000 7f53ed40e640 10 monclient(hunting): _negotiate 2024-03-25T18:27:09.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.777+0000 7f53ed40e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:09.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.777+0000 7f53ed40e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:09.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.777+0000 7f53ed40e640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:27:09.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.777+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e80ada80 con 0x7f53e806e740 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3191572463 0 0) 0x7f53d4003520 con 0x7f53e80ae280 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 >> v1:172.21.15.22:6790/0 conn(0x7f53e8061830 legacy=0x7f53e813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 >> v1:172.21.15.33:6789/0 conn(0x7f53e806e740 legacy=0x7f53e8145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 10 monclient: _finish_hunting 0 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 1 monclient: found mon.noname-a 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e813fcd0 con 0x7f53e80ae280 2024-03-25T18:27:09.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 10 monclient: _finish_auth 0 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 10 monclient: _check_auth_tickets 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53e0001a10 con 0x7f53e80ae280 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53ed40e640 10 monclient: dump: 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:09.784 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:09.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53f0674640 5 monclient: authenticate success, global_id 4569 2024-03-25T18:27:09.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.781+0000 7f53f0674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:09.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53e0001b70 con 0x7f53e80ae280 2024-03-25T18:27:09.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53ed40e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:09.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53ed40e640 1 -- 172.21.15.22:0/822244271 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53e0001f80 con 0x7f53e80ae280 2024-03-25T18:27:09.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53ed40e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:09.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53ed40e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:09.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53ed40e640 10 monclient: dump: 2024-03-25T18:27:09.787 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53f0674640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:09.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.786+0000 7f53f0674640 4 set_mon_vals no callback set 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.787+0000 7f53f0674640 10 monclient: shutdown 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.787+0000 7f53f0674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.787+0000 7f53f0674640 1 -- 172.21.15.22:0/822244271 >> v1:172.21.15.22:6789/0 conn(0x7f53e80ae280 legacy=0x7f53e8145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.787+0000 7f53f0674640 1 -- 172.21.15.22:0/822244271 shutdown_connections 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.787+0000 7f53f0674640 1 -- 172.21.15.22:0/822244271 >> 172.21.15.22:0/822244271 conn(0x7f53e80b0ef0 msgr2=0x7f53e80b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.787+0000 7f53f0674640 1 -- 172.21.15.22:0/822244271 shutdown_connections 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.787+0000 7f53f0674640 1 -- 172.21.15.22:0/822244271 wait complete. 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 10 monclient: build_initial_monmap 2024-03-25T18:27:09.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 10 monclient: monmap: 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 1 librados: starting msgr at 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 1 librados: starting objecter 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 1 Processor -- start 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 1 -- start start 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 1 librados: setting wanted keys 2024-03-25T18:27:09.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 1 librados: calling monclient init 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.788+0000 7f53f0674640 10 monclient: init 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 10 monclient(hunting): picked mon.noname-a con 0x7f53e8061830 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 10 monclient(hunting): picked mon.noname-b con 0x7f53e80ae280 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 10 monclient(hunting): picked mon.noname-c con 0x7f53e8143890 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e8144bf0 con 0x7f53e8061830 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e813f010 con 0x7f53e80ae280 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e8144e40 con 0x7f53e8143890 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:09.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.789+0000 7f53f0674640 10 monclient(hunting): authenticate will time out at 7629.013438s 2024-03-25T18:27:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.802+0000 7f53eec11640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f53e8143890 0x7f53e81e2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:43840/0 (socket says 172.21.15.22:43840) 2024-03-25T18:27:09.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.802+0000 7f53eec11640 1 -- 172.21.15.22:0/1575180778 learned_addr learned my addr 172.21.15.22:0/1575180778 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:09.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.803+0000 7f53deffd640 10 client.?.objecter ms_handle_connect 0x7f53e8143890 2024-03-25T18:27:09.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.803+0000 7f53deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:09.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.804+0000 7f53deffd640 10 client.?.objecter ms_handle_connect 0x7f53e8061830 2024-03-25T18:27:09.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.804+0000 7f53deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:09.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.806+0000 7f53deffd640 10 client.?.objecter ms_handle_connect 0x7f53e80ae280 2024-03-25T18:27:09.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.806+0000 7f53deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:09.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.820+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976330170 0 0) 0x7f53e8144bf0 con 0x7f53e8061830 2024-03-25T18:27:09.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.820+0000 7f53deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:09.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.820+0000 7f53deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:09.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.820+0000 7f53deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:09.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.820+0000 7f53deffd640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:27:09.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.820+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53cc004eb0 con 0x7f53e8061830 2024-03-25T18:27:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.824+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453223652 0 0) 0x7f53e8144e40 con 0x7f53e8143890 2024-03-25T18:27:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.824+0000 7f53deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.824+0000 7f53deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.824+0000 7f53deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.824+0000 7f53deffd640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:27:09.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.824+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e8144bf0 con 0x7f53e8143890 2024-03-25T18:27:09.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.827+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507608873 0 0) 0x7f53e813f010 con 0x7f53e80ae280 2024-03-25T18:27:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.827+0000 7f53deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.827+0000 7f53deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.827+0000 7f53deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.827+0000 7f53deffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:27:09.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.827+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e8144e40 con 0x7f53e80ae280 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.829+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3383948733 0 0) 0x7f53cc004eb0 con 0x7f53e8061830 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.829+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53e813f010 con 0x7f53e8061830 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.829+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53e4001970 con 0x7f53e8061830 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.829+0000 7f53deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.829+0000 7f53deffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.829+0000 7f53deffd640 10 monclient(hunting): dump: 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:09.831 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:09.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.829+0000 7f53deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.835+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655519445 0 0) 0x7f53e8144e40 con 0x7f53e80ae280 2024-03-25T18:27:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.835+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53cc004eb0 con 0x7f53e80ae280 2024-03-25T18:27:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.835+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53d8001830 con 0x7f53e80ae280 2024-03-25T18:27:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.835+0000 7f53deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.835+0000 7f53deffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:09.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.835+0000 7f53deffd640 10 monclient(hunting): dump: 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:09.839 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.845+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2089086790 0 0) 0x7f53e8144bf0 con 0x7f53e8143890 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.845+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53e8144e40 con 0x7f53e8143890 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.845+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53e0001e80 con 0x7f53e8143890 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.845+0000 7f53deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.845+0000 7f53deffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.845+0000 7f53deffd640 10 monclient(hunting): dump: 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.847 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:09.848 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:09.848 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:09.848 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:09.848 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:09.848 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:09.848 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:09.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3726224753 0 0) 0x7f53cc004eb0 con 0x7f53e80ae280 2024-03-25T18:27:09.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 >> v1:172.21.15.22:6790/0 conn(0x7f53e8143890 legacy=0x7f53e81e2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 >> v1:172.21.15.22:6789/0 conn(0x7f53e8061830 legacy=0x7f53e81400a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:09.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:09.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 1 monclient: found mon.b 2024-03-25T18:27:09.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e8146300 con 0x7f53e80ae280 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 10 monclient: _renew_subs 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e8143060 con 0x7f53e80ae280 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 10 monclient: _renew_subs 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:09.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e8140990 con 0x7f53e80ae280 2024-03-25T18:27:09.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.850+0000 7f53f0674640 1 librados: init done 2024-03-25T18:27:09.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.851+0000 7f53f0674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:09.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.851+0000 7f53f0674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:09.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.851+0000 7f53f0674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:09.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:09.851+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e8140c40 con 0x7f53e80ae280 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.129+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53d8001d40 con 0x7f53e80ae280 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.129+0000 7f53deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.129+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53d8004980 con 0x7f53e80ae280 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.129+0000 7f53deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.129+0000 7f53deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.129+0000 7f53ecc0d640 4 set_mon_vals no callback set 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.129+0000 7f53deffd640 10 monclient: dump: 2024-03-25T18:27:10.131 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:10.132 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:10.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.130+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f53d801a9d0 con 0x7f53e80ae280 2024-03-25T18:27:10.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.131+0000 7f53ee410640 10 monclient: get_auth_request con 0x7f53cc06e160 auth_method 0 2024-03-25T18:27:10.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.131+0000 7f53deffd640 10 client.4582.objecter ms_handle_connect 0x7f53cc06e160 2024-03-25T18:27:10.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f53d80546f0 con 0x7f53e80ae280 2024-03-25T18:27:10.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 10 client.4582.objecter ms_dispatch 0x7f53e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:10.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:10.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:10.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:27:10.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53d8050030 con 0x7f53e80ae280 2024-03-25T18:27:10.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:10.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.135+0000 7f53deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:10.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.363+0000 7f53f0674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:10.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.363+0000 7f53f0674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:10.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.363+0000 7f53f0674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:10.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.363+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53e8142b50 con 0x7f53e80ae280 2024-03-25T18:27:10.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.398+0000 7f53deffd640 1 -- 172.21.15.22:0/1575180778 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f53d8052130 con 0x7f53e80ae280 2024-03-25T18:27:10.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.398+0000 7f53deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:10.400 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.398+0000 7f53deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:10.400 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:10.400 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 10 librados: watch_flush enter 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 10 librados: watch_flush exit 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 >> v1:172.21.15.33:6800/215350958 conn(0x7f53cc06e160 legacy=0x7f53cc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 10 monclient: shutdown 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 >> v1:172.21.15.33:6789/0 conn(0x7f53e80ae280 legacy=0x7f53e81def10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 shutdown_connections 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 >> 172.21.15.22:0/1575180778 conn(0x7f53e80b0ef0 msgr2=0x7f53e80b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 shutdown_connections 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.401+0000 7f53f0674640 1 -- 172.21.15.22:0/1575180778 wait complete. 2024-03-25T18:27:10.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:10.402+0000 7f53f0674640 1 librados: shutdown 2024-03-25T18:27:10.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:17.221 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:17.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.333+0000 7f5bed5f6640 10 monclient: get_monmap_and_config 2024-03-25T18:27:17.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.333+0000 7f5bed5f6640 10 monclient: build_initial_monmap 2024-03-25T18:27:17.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.333+0000 7f5bed5f6640 10 monclient: monmap: 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:17.335059+0000 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:17.335059+0000 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.334+0000 7f5bed5f6640 1 Processor -- start 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.334+0000 7f5bed5f6640 1 -- start start 2024-03-25T18:27:17.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.334+0000 7f5bed5f6640 10 monclient: init 2024-03-25T18:27:17.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:17.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:17.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5be80a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:17.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5be813f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:17.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5be81430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:17.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be8142520 con 0x7f5be813f710 2024-03-25T18:27:17.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be80b28a0 con 0x7f5be80a98f0 2024-03-25T18:27:17.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be80b2a30 con 0x7f5be81430c0 2024-03-25T18:27:17.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:17.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.335+0000 7f5bed5f6640 10 monclient(hunting): authenticate will time out at 7636.559109s 2024-03-25T18:27:17.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.349+0000 7f5be67fc640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f5be813f710 0x7f5be81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55484/0 (socket says 172.21.15.22:55484) 2024-03-25T18:27:17.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.349+0000 7f5be67fc640 1 -- 172.21.15.22:0/722874675 learned_addr learned my addr 172.21.15.22:0/722874675 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:17.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.355+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747575197 0 0) 0x7f5be80b28a0 con 0x7f5be80a98f0 2024-03-25T18:27:17.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.355+0000 7f5be5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:17.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.355+0000 7f5be5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:17.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.355+0000 7f5be5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:17.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.355+0000 7f5be5ffb640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:27:17.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.355+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd0003520 con 0x7f5be80a98f0 2024-03-25T18:27:17.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.361+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781505475 0 0) 0x7f5be8142520 con 0x7f5be813f710 2024-03-25T18:27:17.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.361+0000 7f5be5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:17.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.361+0000 7f5be5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:17.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.361+0000 7f5be5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:17.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.361+0000 7f5be5ffb640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:27:17.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.361+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be80b28a0 con 0x7f5be813f710 2024-03-25T18:27:17.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2706416168 0 0) 0x7f5bd0003520 con 0x7f5be80a98f0 2024-03-25T18:27:17.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 >> v1:172.21.15.22:6790/0 conn(0x7f5be81430c0 legacy=0x7f5be814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 >> v1:172.21.15.22:6789/0 conn(0x7f5be813f710 legacy=0x7f5be81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:17.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.363+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be80b44c0 con 0x7f5be80a98f0 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5be5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5be5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5be5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5bd8001690 con 0x7f5be80a98f0 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5bed5f6640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5bed5f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5be5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5be5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5be5ffb640 10 monclient: dump: 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:17.366 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:17.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.364+0000 7f5bed5f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:17.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bd80017f0 con 0x7f5be80a98f0 2024-03-25T18:27:17.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5be5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:17.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5be5ffb640 1 -- 172.21.15.22:0/722874675 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5bd8001c00 con 0x7f5be80a98f0 2024-03-25T18:27:17.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5be5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:17.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5be5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:17.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5be5ffb640 10 monclient: dump: 2024-03-25T18:27:17.372 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5bed5f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:17.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.371+0000 7f5bed5f6640 4 set_mon_vals no callback set 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.372+0000 7f5bed5f6640 10 monclient: shutdown 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.372+0000 7f5bed5f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.372+0000 7f5bed5f6640 1 -- 172.21.15.22:0/722874675 >> v1:172.21.15.33:6789/0 conn(0x7f5be80a98f0 legacy=0x7f5be80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.372+0000 7f5bed5f6640 1 -- 172.21.15.22:0/722874675 shutdown_connections 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.372+0000 7f5bed5f6640 1 -- 172.21.15.22:0/722874675 >> 172.21.15.22:0/722874675 conn(0x7f5be80b7ea0 msgr2=0x7f5be80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.372+0000 7f5bed5f6640 1 -- 172.21.15.22:0/722874675 shutdown_connections 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.372+0000 7f5bed5f6640 1 -- 172.21.15.22:0/722874675 wait complete. 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 10 monclient: build_initial_monmap 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 10 monclient: monmap: 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:17.376 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 1 librados: starting msgr at 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 1 librados: starting objecter 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 1 Processor -- start 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 1 -- start start 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 1 librados: setting wanted keys 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 1 librados: calling monclient init 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.373+0000 7f5bed5f6640 10 monclient: init 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:17.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 10 monclient(hunting): picked mon.noname-c con 0x7f5be80a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 10 monclient(hunting): picked mon.noname-a con 0x7f5be813f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 10 monclient(hunting): picked mon.noname-b con 0x7f5be81430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be8137ab0 con 0x7f5be813f710 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be8138bd0 con 0x7f5be81430c0 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be8138e40 con 0x7f5be80a98f0 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:17.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.374+0000 7f5bed5f6640 10 monclient(hunting): authenticate will time out at 7636.598163s 2024-03-25T18:27:17.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.386+0000 7f5be6ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5be80a98f0 0x7f5be8137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39762/0 (socket says 172.21.15.22:39762) 2024-03-25T18:27:17.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.386+0000 7f5be6ffd640 1 -- 172.21.15.22:0/2478092427 learned_addr learned my addr 172.21.15.22:0/2478092427 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:17.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.388+0000 7f5bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f5be80a98f0 2024-03-25T18:27:17.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.388+0000 7f5bcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:17.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.395+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547623730 0 0) 0x7f5be8138e40 con 0x7f5be80a98f0 2024-03-25T18:27:17.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.395+0000 7f5bcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:17.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.395+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:17.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.395+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:17.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.395+0000 7f5bcf7fe640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:27:17.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.395+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd4004eb0 con 0x7f5be80a98f0 2024-03-25T18:27:17.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.396+0000 7f5bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f5be813f710 2024-03-25T18:27:17.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.396+0000 7f5bcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:17.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.400+0000 7f5bcf7fe640 10 client.?.objecter ms_handle_connect 0x7f5be81430c0 2024-03-25T18:27:17.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.400+0000 7f5bcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:17.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3484422565 0 0) 0x7f5bd4004eb0 con 0x7f5be80a98f0 2024-03-25T18:27:17.412 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5be8138e40 con 0x7f5be80a98f0 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5bd8001bc0 con 0x7f5be80a98f0 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:17.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:17.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374664525 0 0) 0x7f5be8138bd0 con 0x7f5be81430c0 2024-03-25T18:27:17.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:17.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:17.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:17.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:27:17.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.411+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bd4004eb0 con 0x7f5be81430c0 2024-03-25T18:27:17.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.415+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658302241 0 0) 0x7f5be8137ab0 con 0x7f5be813f710 2024-03-25T18:27:17.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.415+0000 7f5bcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:17.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.415+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:17.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.415+0000 7f5bcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:17.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.415+0000 7f5bcf7fe640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:27:17.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.415+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be8138bd0 con 0x7f5be813f710 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.428+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 90151902 0 0) 0x7f5be8138bd0 con 0x7f5be813f710 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.428+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5be8137ab0 con 0x7f5be813f710 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.428+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5bdc001930 con 0x7f5be813f710 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.428+0000 7f5bcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.428+0000 7f5bcf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.428+0000 7f5bcf7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:17.430 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:17.431 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:17.431 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:17.431 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:17.431 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:17.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4155574111 0 0) 0x7f5be8138e40 con 0x7f5be80a98f0 2024-03-25T18:27:17.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 >> v1:172.21.15.33:6789/0 conn(0x7f5be81430c0 legacy=0x7f5be81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 >> v1:172.21.15.22:6789/0 conn(0x7f5be813f710 legacy=0x7f5be81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 1 monclient: found mon.c 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be8139120 con 0x7f5be80a98f0 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 10 monclient: _renew_subs 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be8139400 con 0x7f5be80a98f0 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:17.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 10 monclient: _renew_subs 2024-03-25T18:27:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:27:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be80b7880 con 0x7f5be80a98f0 2024-03-25T18:27:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.436+0000 7f5bed5f6640 1 librados: init done 2024-03-25T18:27:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.437+0000 7f5bed5f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.437+0000 7f5bed5f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.437+0000 7f5bed5f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:27:17.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.437+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be8137810 con 0x7f5be80a98f0 2024-03-25T18:27:17.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.748+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5bd80011f0 con 0x7f5be80a98f0 2024-03-25T18:27:17.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.748+0000 7f5bcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.748+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5bd8003a40 con 0x7f5be80a98f0 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.748+0000 7f5bcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.748+0000 7f5bcf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.748+0000 7f5bcf7fe640 10 monclient: dump: 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:17.750 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:17.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.748+0000 7f5be57fa640 4 set_mon_vals no callback set 2024-03-25T18:27:17.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.749+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f5bd8003c60 con 0x7f5be80a98f0 2024-03-25T18:27:17.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.749+0000 7f5be67fc640 10 monclient: get_auth_request con 0x7f5bd406df80 auth_method 0 2024-03-25T18:27:17.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.750+0000 7f5bcf7fe640 10 client.4592.objecter ms_handle_connect 0x7f5bd406df80 2024-03-25T18:27:17.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f5bd8054970 con 0x7f5be80a98f0 2024-03-25T18:27:17.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 10 client.4592.objecter ms_dispatch 0x7f5be8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:17.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:17.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:17.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-25T18:27:17.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5bd80502b0 con 0x7f5be80a98f0 2024-03-25T18:27:17.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:17.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.756+0000 7f5bcf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:17.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.989+0000 7f5bed5f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.989+0000 7f5bed5f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:17.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.989+0000 7f5bed5f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:27:17.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:17.989+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5be80627b0 con 0x7f5be80a98f0 2024-03-25T18:27:18.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.124+0000 7f5bcf7fe640 1 -- 172.21.15.22:0/2478092427 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f5bd80523b0 con 0x7f5be80a98f0 2024-03-25T18:27:18.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.124+0000 7f5bcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:18.125 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.124+0000 7f5bcf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:18.126 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:18.126 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 10 librados: watch_flush enter 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 10 librados: watch_flush exit 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 10 client.4592.objecter successfully canceled tick 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 >> v1:172.21.15.33:6800/215350958 conn(0x7f5bd406df80 legacy=0x7f5bd4070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 10 monclient: shutdown 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:18.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 >> v1:172.21.15.22:6790/0 conn(0x7f5be80a98f0 legacy=0x7f5be8137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:18.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 shutdown_connections 2024-03-25T18:27:18.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 >> 172.21.15.22:0/2478092427 conn(0x7f5be80b7ea0 msgr2=0x7f5be8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:18.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 shutdown_connections 2024-03-25T18:27:18.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 1 -- 172.21.15.22:0/2478092427 wait complete. 2024-03-25T18:27:18.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:18.126+0000 7f5bed5f6640 1 librados: shutdown 2024-03-25T18:27:18.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:18.917 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:21.579 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.929 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:21.590 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.965 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:21.627 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.968 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:21.630 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.979 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:21.597 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.980 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:21.641 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.989 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:21.607 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:19.010 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:21.672 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:19.033 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:21.651 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:24.949 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:25.063 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.061+0000 7f4e9214d640 10 monclient: get_monmap_and_config 2024-03-25T18:27:25.063 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.061+0000 7f4e9214d640 10 monclient: build_initial_monmap 2024-03-25T18:27:25.063 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.061+0000 7f4e9214d640 10 monclient: monmap: 2024-03-25T18:27:25.063 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:25.063 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:25.063254+0000 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:25.063254+0000 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:25.064 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:25.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.062+0000 7f4e9214d640 1 Processor -- start 2024-03-25T18:27:25.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 1 -- start start 2024-03-25T18:27:25.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient: init 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient(hunting): picked mon.noname-c con 0x7f4e8c0705b0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8c1402d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8c149ae0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c0718c0 con 0x7f4e8c1402d0 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c0714c0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c072b50 con 0x7f4e8c149ae0 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:25.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.063+0000 7f4e9214d640 10 monclient(hunting): authenticate will time out at 7644.287308s 2024-03-25T18:27:25.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.079+0000 7f4e8b7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f4e8c0705b0 0x7f4e8c13d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:55678/0 (socket says 172.21.15.22:55678) 2024-03-25T18:27:25.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.079+0000 7f4e8b7fe640 1 -- 172.21.15.22:0/287772685 learned_addr learned my addr 172.21.15.22:0/287772685 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:25.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.094+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056832531 0 0) 0x7f4e8c0718c0 con 0x7f4e8c1402d0 2024-03-25T18:27:25.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.094+0000 7f4e8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.094+0000 7f4e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.094+0000 7f4e8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.094+0000 7f4e8a7fc640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:27:25.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.094+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e74003520 con 0x7f4e8c1402d0 2024-03-25T18:27:25.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.095+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103270354 0 0) 0x7f4e8c072b50 con 0x7f4e8c149ae0 2024-03-25T18:27:25.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.095+0000 7f4e8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.095+0000 7f4e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.095+0000 7f4e8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.095+0000 7f4e8a7fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:27:25.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.095+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c0718c0 con 0x7f4e8c149ae0 2024-03-25T18:27:25.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.100+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20336912 0 0) 0x7f4e8c0714c0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.100+0000 7f4e8a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.100+0000 7f4e8a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.100+0000 7f4e8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.100+0000 7f4e8a7fc640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:27:25.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.100+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c072b50 con 0x7f4e8c0705b0 2024-03-25T18:27:25.103 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975761992 0 0) 0x7f4e74003520 con 0x7f4e8c1402d0 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 >> v1:172.21.15.22:6790/0 conn(0x7f4e8c149ae0 legacy=0x7f4e8c14bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 >> v1:172.21.15.33:6789/0 conn(0x7f4e8c0705b0 legacy=0x7f4e8c13d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c13dd10 con 0x7f4e8c1402d0 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:25.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e80001690 con 0x7f4e8c1402d0 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e8a7fc640 10 monclient: dump: 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:25.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e9214d640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:27:25.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.101+0000 7f4e9214d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:25.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.105+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e800017f0 con 0x7f4e8c1402d0 2024-03-25T18:27:25.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.105+0000 7f4e8a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:25.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.105+0000 7f4e9214d640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:25.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.105+0000 7f4e9214d640 4 set_mon_vals no callback set 2024-03-25T18:27:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 10 monclient: shutdown 2024-03-25T18:27:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 1 -- 172.21.15.22:0/287772685 >> v1:172.21.15.22:6789/0 conn(0x7f4e8c1402d0 legacy=0x7f4e8c140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e8a7fc640 1 -- 172.21.15.22:0/287772685 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e80001c00 con 0x7f4e8c1402d0 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e8a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 1 -- 172.21.15.22:0/287772685 shutdown_connections 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 1 -- 172.21.15.22:0/287772685 >> 172.21.15.22:0/287772685 conn(0x7f4e8c0b5d10 msgr2=0x7f4e8c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 1 -- 172.21.15.22:0/287772685 shutdown_connections 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 1 -- 172.21.15.22:0/287772685 wait complete. 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 10 monclient: build_initial_monmap 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.106+0000 7f4e9214d640 10 monclient: monmap: 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:25.109 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.107+0000 7f4e9214d640 1 librados: starting msgr at 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.107+0000 7f4e9214d640 1 librados: starting objecter 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.107+0000 7f4e9214d640 1 Processor -- start 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.107+0000 7f4e9214d640 1 -- start start 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.107+0000 7f4e9214d640 1 librados: setting wanted keys 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.107+0000 7f4e9214d640 1 librados: calling monclient init 2024-03-25T18:27:25.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.107+0000 7f4e9214d640 10 monclient: init 2024-03-25T18:27:25.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:25.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:25.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8c0705b0 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:25.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8c1402d0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:25.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 10 monclient(hunting): picked mon.noname-c con 0x7f4e8c149ae0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:25.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c1d9830 con 0x7f4e8c0705b0 2024-03-25T18:27:25.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c1d8c00 con 0x7f4e8c1402d0 2024-03-25T18:27:25.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8c142eb0 con 0x7f4e8c149ae0 2024-03-25T18:27:25.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:25.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.108+0000 7f4e9214d640 10 monclient(hunting): authenticate will time out at 7644.331903s 2024-03-25T18:27:25.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.118+0000 7f4e8bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4e8c149ae0 0x7f4e8c148d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53540/0 (socket says 172.21.15.22:53540) 2024-03-25T18:27:25.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.118+0000 7f4e8bfff640 1 -- 172.21.15.22:0/3415203806 learned_addr learned my addr 172.21.15.22:0/3415203806 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:25.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.120+0000 7f4e73fff640 10 client.?.objecter ms_handle_connect 0x7f4e8c0705b0 2024-03-25T18:27:25.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.120+0000 7f4e73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:25.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.121+0000 7f4e73fff640 10 client.?.objecter ms_handle_connect 0x7f4e8c149ae0 2024-03-25T18:27:25.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.121+0000 7f4e73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:25.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.127+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649052986 0 0) 0x7f4e8c1d9830 con 0x7f4e8c0705b0 2024-03-25T18:27:25.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.127+0000 7f4e73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.127+0000 7f4e73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.127+0000 7f4e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.127+0000 7f4e73fff640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:27:25.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.127+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e78004eb0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.130+0000 7f4e73fff640 10 client.?.objecter ms_handle_connect 0x7f4e8c1402d0 2024-03-25T18:27:25.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.130+0000 7f4e73fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:25.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.131+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3555854305 0 0) 0x7f4e8c142eb0 con 0x7f4e8c149ae0 2024-03-25T18:27:25.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.131+0000 7f4e73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.131+0000 7f4e73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.131+0000 7f4e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.131+0000 7f4e73fff640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:27:25.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.131+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c1d9830 con 0x7f4e8c149ae0 2024-03-25T18:27:25.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.141+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866945941 0 0) 0x7f4e78004eb0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.142+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e8c142eb0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.142+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e7c0018f0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.142+0000 7f4e73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:25.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.142+0000 7f4e73fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:25.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.142+0000 7f4e73fff640 10 monclient(hunting): dump: 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:25.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.142+0000 7f4e73fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:25.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.144+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427766167 0 0) 0x7f4e8c1d8c00 con 0x7f4e8c1402d0 2024-03-25T18:27:25.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.144+0000 7f4e73fff640 10 monclient(hunting): _negotiate 2024-03-25T18:27:25.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.144+0000 7f4e73fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:25.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.144+0000 7f4e73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:25.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.144+0000 7f4e73fff640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:27:25.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.144+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e78004eb0 con 0x7f4e8c1402d0 2024-03-25T18:27:25.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.170+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1344244989 0 0) 0x7f4e8c1d9830 con 0x7f4e8c149ae0 2024-03-25T18:27:25.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.170+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e8c1d8c00 con 0x7f4e8c149ae0 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.170+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e84001bd0 con 0x7f4e8c149ae0 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.170+0000 7f4e73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.170+0000 7f4e73fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.170+0000 7f4e73fff640 10 monclient(hunting): dump: 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:25.172 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:25.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3723887007 0 0) 0x7f4e8c142eb0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 >> v1:172.21.15.22:6790/0 conn(0x7f4e8c149ae0 legacy=0x7f4e8c148d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 >> v1:172.21.15.33:6789/0 conn(0x7f4e8c1402d0 legacy=0x7f4e8c1427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 10 monclient: _finish_hunting 0 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 1 monclient: found mon.a 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.171+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c141cc0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e73fff640 10 monclient: _finish_auth 0 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e73fff640 10 monclient: _check_auth_tickets 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 monclient: _renew_subs 2024-03-25T18:27:25.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:25.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8c143110 con 0x7f4e8c0705b0 2024-03-25T18:27:25.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:25.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 monclient: _renew_subs 2024-03-25T18:27:25.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:25.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8c13e1b0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 1 librados: init done 2024-03-25T18:27:25.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:25.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:25.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:25.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.172+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8c13d9c0 con 0x7f4e8c0705b0 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.486+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e7c001d40 con 0x7f4e8c0705b0 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.486+0000 7f4e73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.486+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f4e7c004a80 con 0x7f4e8c0705b0 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.486+0000 7f4e73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.486+0000 7f4e73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.486+0000 7f4e73fff640 10 monclient: dump: 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:25.488 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:25.489 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:25.489 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:25.489 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:25.489 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:25.489 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:25.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.486+0000 7f4e89ffb640 4 set_mon_vals no callback set 2024-03-25T18:27:25.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.487+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f4e7c01ab10 con 0x7f4e8c0705b0 2024-03-25T18:27:25.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.488+0000 7f4e8affd640 10 monclient: get_auth_request con 0x7f4e7806ddf0 auth_method 0 2024-03-25T18:27:25.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.488+0000 7f4e73fff640 10 client.4584.objecter ms_handle_connect 0x7f4e7806ddf0 2024-03-25T18:27:25.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f4e7c054740 con 0x7f4e8c0705b0 2024-03-25T18:27:25.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 10 client.4584.objecter ms_dispatch 0x7f4e8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:25.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 3 client.4584.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:25.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 3 client.4584.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:25.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-25T18:27:25.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e7c050080 con 0x7f4e8c0705b0 2024-03-25T18:27:25.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:25.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.493+0000 7f4e73fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:25.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.740+0000 7f4e9214d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:25.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.740+0000 7f4e9214d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:25.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.740+0000 7f4e9214d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:27:25.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.740+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e8c141e50 con 0x7f4e8c0705b0 2024-03-25T18:27:25.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.788+0000 7f4e73fff640 1 -- 172.21.15.22:0/3415203806 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f4e7c052180 con 0x7f4e8c0705b0 2024-03-25T18:27:25.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.788+0000 7f4e73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:25.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.788+0000 7f4e73fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:25.790 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:25.790 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:25.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 10 librados: watch_flush enter 2024-03-25T18:27:25.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 10 librados: watch_flush exit 2024-03-25T18:27:25.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-25T18:27:25.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 10 client.4584.objecter successfully canceled tick 2024-03-25T18:27:25.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 >> v1:172.21.15.33:6800/215350958 conn(0x7f4e7806ddf0 legacy=0x7f4e780702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 10 monclient: shutdown 2024-03-25T18:27:25.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:25.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 >> v1:172.21.15.22:6789/0 conn(0x7f4e8c0705b0 legacy=0x7f4e8c1d9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:25.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 shutdown_connections 2024-03-25T18:27:25.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.790+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 >> 172.21.15.22:0/3415203806 conn(0x7f4e8c0b5d10 msgr2=0x7f4e8c13c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:25.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.791+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 shutdown_connections 2024-03-25T18:27:25.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.791+0000 7f4e9214d640 1 -- 172.21.15.22:0/3415203806 wait complete. 2024-03-25T18:27:25.793 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:25.791+0000 7f4e9214d640 1 librados: shutdown 2024-03-25T18:27:25.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:32.610 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:32.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.723+0000 7fced230e640 10 monclient: get_monmap_and_config 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.723+0000 7fced230e640 10 monclient: build_initial_monmap 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.723+0000 7fced230e640 10 monclient: monmap: 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:32.724712+0000 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:32.724712+0000 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:32.725 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.724+0000 7fced230e640 1 Processor -- start 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.724+0000 7fced230e640 1 -- start start 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.724+0000 7fced230e640 10 monclient: init 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 10 monclient(hunting): picked mon.noname-c con 0x7fcecc0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 10 monclient(hunting): picked mon.noname-b con 0x7fcecc13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 10 monclient(hunting): picked mon.noname-a con 0x7fcecc1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc142520 con 0x7fcecc1430c0 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc0b28a0 con 0x7fcecc0a98f0 2024-03-25T18:27:32.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc0b2a30 con 0x7fcecc13f710 2024-03-25T18:27:32.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:32.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.725+0000 7fced230e640 10 monclient(hunting): authenticate will time out at 7651.948757s 2024-03-25T18:27:32.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.730+0000 7fcecb7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fcecc13f710 0x7fcecc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58006/0 (socket says 172.21.15.22:58006) 2024-03-25T18:27:32.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.730+0000 7fcecb7fe640 1 -- 172.21.15.22:0/727125682 learned_addr learned my addr 172.21.15.22:0/727125682 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:32.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.739+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040375210 0 0) 0x7fcecc0b28a0 con 0x7fcecc0a98f0 2024-03-25T18:27:32.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.740+0000 7fcecaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:32.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.740+0000 7fcecaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:32.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.740+0000 7fcecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:32.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.740+0000 7fcecaffd640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:27:32.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.740+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb4003520 con 0x7fcecc0a98f0 2024-03-25T18:27:32.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.745+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416200349 0 0) 0x7fcecc0b2a30 con 0x7fcecc13f710 2024-03-25T18:27:32.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.745+0000 7fcecaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:32.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.745+0000 7fcecaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:32.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.745+0000 7fcecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:32.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.745+0000 7fcecaffd640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:27:32.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.745+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc0b28a0 con 0x7fcecc13f710 2024-03-25T18:27:32.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.756+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046350693 0 0) 0x7fcecc142520 con 0x7fcecc1430c0 2024-03-25T18:27:32.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.756+0000 7fcecaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:32.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.756+0000 7fcecaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:32.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.756+0000 7fcecaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:32.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.756+0000 7fcecaffd640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:27:32.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.756+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc0b2a30 con 0x7fcecc1430c0 2024-03-25T18:27:32.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.759+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3337797520 0 0) 0x7fceb4003520 con 0x7fcecc0a98f0 2024-03-25T18:27:32.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.759+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 >> v1:172.21.15.22:6790/0 conn(0x7fcecc13f710 legacy=0x7fcecc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.761 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.759+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 >> v1:172.21.15.22:6789/0 conn(0x7fcecc1430c0 legacy=0x7fcecc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.759+0000 7fcecaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.759+0000 7fcecaffd640 1 monclient: found mon.noname-c 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc0b44c0 con 0x7fcecc0a98f0 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcec4001a10 con 0x7fcecc0a98f0 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fcecaffd640 10 monclient: dump: 2024-03-25T18:27:32.762 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fced230e640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:27:32.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.760+0000 7fced230e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:32.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec4001b70 con 0x7fcecc0a98f0 2024-03-25T18:27:32.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fcecaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:32.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fcecaffd640 1 -- 172.21.15.22:0/727125682 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcec4001f80 con 0x7fcecc0a98f0 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fced230e640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fced230e640 4 set_mon_vals no callback set 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fcecaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fcecaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.770+0000 7fcecaffd640 10 monclient: dump: 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:32.772 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:32.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.771+0000 7fced230e640 10 monclient: shutdown 2024-03-25T18:27:32.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.771+0000 7fced230e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:32.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.771+0000 7fced230e640 1 -- 172.21.15.22:0/727125682 >> v1:172.21.15.33:6789/0 conn(0x7fcecc0a98f0 legacy=0x7fcecc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.771+0000 7fced230e640 1 -- 172.21.15.22:0/727125682 shutdown_connections 2024-03-25T18:27:32.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.771+0000 7fced230e640 1 -- 172.21.15.22:0/727125682 >> 172.21.15.22:0/727125682 conn(0x7fcecc0b7ea0 msgr2=0x7fcecc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:32.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.771+0000 7fced230e640 1 -- 172.21.15.22:0/727125682 shutdown_connections 2024-03-25T18:27:32.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.771+0000 7fced230e640 1 -- 172.21.15.22:0/727125682 wait complete. 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 10 monclient: build_initial_monmap 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 10 monclient: monmap: 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 1 librados: starting msgr at 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 1 librados: starting objecter 2024-03-25T18:27:32.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 1 Processor -- start 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 1 -- start start 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 1 librados: setting wanted keys 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 1 librados: calling monclient init 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.772+0000 7fced230e640 10 monclient: init 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 10 monclient(hunting): picked mon.noname-c con 0x7fcecc0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 10 monclient(hunting): picked mon.noname-b con 0x7fcecc13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 10 monclient(hunting): picked mon.noname-a con 0x7fcecc1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc139870 con 0x7fcecc1430c0 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc1d51f0 con 0x7fcecc13f710 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcecc138830 con 0x7fcecc0a98f0 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:32.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.773+0000 7fced230e640 10 monclient(hunting): authenticate will time out at 7651.997202s 2024-03-25T18:27:32.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.779+0000 7fcecb7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fcecc13f710 0x7fcecc1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:60348/0 (socket says 172.21.15.22:60348) 2024-03-25T18:27:32.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.779+0000 7fcecb7fe640 1 -- 172.21.15.22:0/909342016 learned_addr learned my addr 172.21.15.22:0/909342016 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:32.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.780+0000 7fcec8ff9640 10 client.?.objecter ms_handle_connect 0x7fcecc13f710 2024-03-25T18:27:32.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.780+0000 7fcec8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:32.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.789+0000 7fcec8ff9640 10 client.?.objecter ms_handle_connect 0x7fcecc0a98f0 2024-03-25T18:27:32.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.789+0000 7fcec8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:32.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.791+0000 7fcec8ff9640 10 client.?.objecter ms_handle_connect 0x7fcecc1430c0 2024-03-25T18:27:32.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.791+0000 7fcec8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:32.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.805+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788449567 0 0) 0x7fcecc1d51f0 con 0x7fcecc13f710 2024-03-25T18:27:32.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.805+0000 7fcec8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:32.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.805+0000 7fcec8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:32.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.805+0000 7fcec8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:32.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.805+0000 7fcec8ff9640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:27:32.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.805+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb8004eb0 con 0x7fcecc13f710 2024-03-25T18:27:32.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.810+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398210017 0 0) 0x7fcecc139870 con 0x7fcecc1430c0 2024-03-25T18:27:32.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.810+0000 7fcec8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:32.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.810+0000 7fcec8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:32.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.810+0000 7fcec8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:32.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.810+0000 7fcec8ff9640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:27:32.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.810+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc1d51f0 con 0x7fcecc1430c0 2024-03-25T18:27:32.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.821+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350850453 0 0) 0x7fcecc138830 con 0x7fcecc0a98f0 2024-03-25T18:27:32.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.821+0000 7fcec8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:27:32.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.821+0000 7fcec8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:32.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.821+0000 7fcec8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:32.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.821+0000 7fcec8ff9640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:27:32.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.821+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc139870 con 0x7fcecc0a98f0 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.827+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 263714454 0 0) 0x7fceb8004eb0 con 0x7fcecc13f710 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.827+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcecc138830 con 0x7fcecc13f710 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.827+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcec00019f0 con 0x7fcecc13f710 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.827+0000 7fcec8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.827+0000 7fcec8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.827+0000 7fcec8ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:32.829 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:32.830 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:32.830 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:32.830 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:32.830 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:32.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.827+0000 7fcec8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.834+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038373071 0 0) 0x7fcecc1d51f0 con 0x7fcecc1430c0 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.834+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fceb8004eb0 con 0x7fcecc1430c0 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.835+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcebc0017d0 con 0x7fcecc1430c0 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.835+0000 7fcec8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.835+0000 7fcec8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.835+0000 7fcec8ff9640 10 monclient(hunting): dump: 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.836 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:32.837 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:32.837 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:32.837 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:32.837 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:32.837 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:32.837 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 459559817 0 0) 0x7fcecc138830 con 0x7fcecc13f710 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 >> v1:172.21.15.22:6790/0 conn(0x7fcecc0a98f0 legacy=0x7fcecc138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 >> v1:172.21.15.22:6789/0 conn(0x7fcecc1430c0 legacy=0x7fcecc1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 1 monclient: found mon.b 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc1d77a0 con 0x7fcecc13f710 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fcec8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:27:32.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 10 monclient: _renew_subs 2024-03-25T18:27:32.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:32.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcecc1d83d0 con 0x7fcecc13f710 2024-03-25T18:27:32.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:32.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 10 monclient: _renew_subs 2024-03-25T18:27:32.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:32.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcecc138c80 con 0x7fcecc13f710 2024-03-25T18:27:32.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.837+0000 7fced230e640 1 librados: init done 2024-03-25T18:27:32.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.838+0000 7fced230e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:32.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.838+0000 7fced230e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:32.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.838+0000 7fced230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:32.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:32.838+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcecc0b3e80 con 0x7fcecc13f710 2024-03-25T18:27:33.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.118+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec00047e0 con 0x7fcecc13f710 2024-03-25T18:27:33.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.118+0000 7fcec8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:33.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.118+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fcec0004af0 con 0x7fcecc13f710 2024-03-25T18:27:33.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.118+0000 7fcec8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:33.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.118+0000 7fcec8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.118+0000 7fcec8ff9640 10 monclient: dump: 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:33.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.118+0000 7fceca7fc640 4 set_mon_vals no callback set 2024-03-25T18:27:33.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.119+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fcec001ab00 con 0x7fcecc13f710 2024-03-25T18:27:33.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.120+0000 7fcecbfff640 10 monclient: get_auth_request con 0x7fceb806df70 auth_method 0 2024-03-25T18:27:33.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.120+0000 7fcec8ff9640 10 client.4600.objecter ms_handle_connect 0x7fceb806df70 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fcec00548b0 con 0x7fcecc13f710 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 10 client.4600.objecter ms_dispatch 0x7fcecc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcec00501f0 con 0x7fcecc13f710 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:33.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.131+0000 7fcec8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:33.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.351+0000 7fced230e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:33.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.351+0000 7fced230e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:33.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.351+0000 7fced230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:33.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.351+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcecc074cc0 con 0x7fcecc13f710 2024-03-25T18:27:33.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.370+0000 7fcec8ff9640 1 -- 172.21.15.22:0/909342016 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fcec00522f0 con 0x7fcecc13f710 2024-03-25T18:27:33.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.371+0000 7fcec8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:33.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.371+0000 7fcec8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:33.373 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:33.373 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 10 librados: watch_flush enter 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 10 librados: watch_flush exit 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 10 client.4600.objecter successfully canceled tick 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 >> v1:172.21.15.33:6800/215350958 conn(0x7fceb806df70 legacy=0x7fceb8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 10 monclient: shutdown 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 >> v1:172.21.15.33:6789/0 conn(0x7fcecc13f710 legacy=0x7fcecc1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 shutdown_connections 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 >> 172.21.15.22:0/909342016 conn(0x7fcecc0b7ea0 msgr2=0x7fcecc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:33.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 shutdown_connections 2024-03-25T18:27:33.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 1 -- 172.21.15.22:0/909342016 wait complete. 2024-03-25T18:27:33.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:33.373+0000 7fced230e640 1 librados: shutdown 2024-03-25T18:27:33.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:38.817 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:41.478 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.822 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:41.484 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.866 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:08:41.528 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.875 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:41.493 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.886 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:41.504 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.901 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:41.563 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.916 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:41.577 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.916 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:08:41.534 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.947 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:08:41.608 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:40.193 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.304+0000 7ff8cabb5640 10 monclient: get_monmap_and_config 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.304+0000 7ff8cabb5640 10 monclient: build_initial_monmap 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.304+0000 7ff8cabb5640 10 monclient: monmap: 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:40.305772+0000 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:40.305772+0000 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:40.306 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.305+0000 7ff8cabb5640 1 Processor -- start 2024-03-25T18:27:40.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.305+0000 7ff8cabb5640 1 -- start start 2024-03-25T18:27:40.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.305+0000 7ff8cabb5640 10 monclient: init 2024-03-25T18:27:40.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:40.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:40.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8c40b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:40.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 10 monclient(hunting): picked mon.noname-b con 0x7ff8c40b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:40.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 10 monclient(hunting): picked mon.noname-c con 0x7ff8c414be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:40.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c40af940 con 0x7ff8c40b4d60 2024-03-25T18:27:40.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c40b0a70 con 0x7ff8c414be90 2024-03-25T18:27:40.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c40b0d50 con 0x7ff8c40b1730 2024-03-25T18:27:40.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:40.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.306+0000 7ff8cabb5640 10 monclient(hunting): authenticate will time out at 7659.529994s 2024-03-25T18:27:40.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.317+0000 7ff8c9152640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7ff8c414be90 0x7ff8c414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:47704/0 (socket says 172.21.15.22:47704) 2024-03-25T18:27:40.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.317+0000 7ff8c9152640 1 -- 172.21.15.22:0/1520611236 learned_addr learned my addr 172.21.15.22:0/1520611236 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:40.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.330+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043970629 0 0) 0x7ff8c40af940 con 0x7ff8c40b4d60 2024-03-25T18:27:40.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.330+0000 7ff8c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:40.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.330+0000 7ff8c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:40.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.330+0000 7ff8c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:40.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.330+0000 7ff8c37fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:27:40.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.330+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8b4003520 con 0x7ff8c40b4d60 2024-03-25T18:27:40.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.334+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3312712697 0 0) 0x7ff8c40b0a70 con 0x7ff8c414be90 2024-03-25T18:27:40.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.334+0000 7ff8c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:40.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.334+0000 7ff8c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:40.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.334+0000 7ff8c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:40.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.334+0000 7ff8c37fe640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:27:40.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.334+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c40af940 con 0x7ff8c414be90 2024-03-25T18:27:40.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.335+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598912691 0 0) 0x7ff8c40b0d50 con 0x7ff8c40b1730 2024-03-25T18:27:40.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.335+0000 7ff8c37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:40.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.335+0000 7ff8c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:40.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.335+0000 7ff8c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:40.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.335+0000 7ff8c37fe640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:27:40.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.335+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c40b0a70 con 0x7ff8c40b1730 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179525220 0 0) 0x7ff8c40af940 con 0x7ff8c414be90 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 >> v1:172.21.15.22:6790/0 conn(0x7ff8c40b1730 legacy=0x7ff8c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 >> v1:172.21.15.22:6789/0 conn(0x7ff8c40b4d60 legacy=0x7ff8c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 1 monclient: found mon.noname-c 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c40b1030 con 0x7ff8c414be90 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8bc0019f0 con 0x7ff8c414be90 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:40.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8c37fe640 10 monclient: dump: 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8cabb5640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:27:40.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.341+0000 7ff8cabb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:40.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.356+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8bc001b50 con 0x7ff8c414be90 2024-03-25T18:27:40.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.356+0000 7ff8c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:40.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.356+0000 7ff8c37fe640 1 -- 172.21.15.22:0/1520611236 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8bc001f60 con 0x7ff8c414be90 2024-03-25T18:27:40.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.356+0000 7ff8c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:40.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.356+0000 7ff8c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:40.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.356+0000 7ff8c37fe640 10 monclient: dump: 2024-03-25T18:27:40.358 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.357+0000 7ff8cabb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:40.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.357+0000 7ff8cabb5640 4 set_mon_vals no callback set 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 10 monclient: shutdown 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 1 -- 172.21.15.22:0/1520611236 >> v1:172.21.15.33:6789/0 conn(0x7ff8c414be90 legacy=0x7ff8c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 1 -- 172.21.15.22:0/1520611236 shutdown_connections 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 1 -- 172.21.15.22:0/1520611236 >> 172.21.15.22:0/1520611236 conn(0x7ff8c40617a0 msgr2=0x7ff8c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 1 -- 172.21.15.22:0/1520611236 shutdown_connections 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 1 -- 172.21.15.22:0/1520611236 wait complete. 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 10 monclient: build_initial_monmap 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.358+0000 7ff8cabb5640 10 monclient: monmap: 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:40.361 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.359+0000 7ff8cabb5640 1 librados: starting msgr at 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.359+0000 7ff8cabb5640 1 librados: starting objecter 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.359+0000 7ff8cabb5640 1 Processor -- start 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.359+0000 7ff8cabb5640 1 -- start start 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.359+0000 7ff8cabb5640 1 librados: setting wanted keys 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.359+0000 7ff8cabb5640 1 librados: calling monclient init 2024-03-25T18:27:40.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.359+0000 7ff8cabb5640 10 monclient: init 2024-03-25T18:27:40.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:40.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:40.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 10 monclient(hunting): picked mon.noname-c con 0x7ff8c40b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:40.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 10 monclient(hunting): picked mon.noname-b con 0x7ff8c40b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:40.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 10 monclient(hunting): picked mon.noname-a con 0x7ff8c414be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:40.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c4141ff0 con 0x7ff8c414be90 2024-03-25T18:27:40.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c41422d0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8c4145630 con 0x7ff8c40b1730 2024-03-25T18:27:40.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:40.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.360+0000 7ff8cabb5640 10 monclient(hunting): authenticate will time out at 7659.584077s 2024-03-25T18:27:40.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.372+0000 7ff8c8951640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff8c40b1730 0x7ff8c4140480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:32994/0 (socket says 172.21.15.22:32994) 2024-03-25T18:27:40.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.372+0000 7ff8c8951640 1 -- 172.21.15.22:0/2481551334 learned_addr learned my addr 172.21.15.22:0/2481551334 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:40.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.377+0000 7ff8c17fa640 10 client.?.objecter ms_handle_connect 0x7ff8c40b1730 2024-03-25T18:27:40.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.377+0000 7ff8c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:40.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.379+0000 7ff8c17fa640 10 client.?.objecter ms_handle_connect 0x7ff8c414be90 2024-03-25T18:27:40.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.379+0000 7ff8c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:40.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.379+0000 7ff8c17fa640 10 client.?.objecter ms_handle_connect 0x7ff8c40b4d60 2024-03-25T18:27:40.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.379+0000 7ff8c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:40.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.386+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482063490 0 0) 0x7ff8c4141ff0 con 0x7ff8c414be90 2024-03-25T18:27:40.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.386+0000 7ff8c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:40.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.386+0000 7ff8c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:40.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.386+0000 7ff8c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:40.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.386+0000 7ff8c17fa640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:27:40.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.386+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8a8004eb0 con 0x7ff8c414be90 2024-03-25T18:27:40.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.390+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680701760 0 0) 0x7ff8c41422d0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.390+0000 7ff8c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:40.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.390+0000 7ff8c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:40.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.390+0000 7ff8c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:40.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.390+0000 7ff8c17fa640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:27:40.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.390+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c4141ff0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.393+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634111863 0 0) 0x7ff8c4145630 con 0x7ff8c40b1730 2024-03-25T18:27:40.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.393+0000 7ff8c17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:40.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.393+0000 7ff8c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:40.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.393+0000 7ff8c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:40.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.393+0000 7ff8c17fa640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:27:40.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.393+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8c41422d0 con 0x7ff8c40b1730 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.415+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3591701923 0 0) 0x7ff8c4141ff0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.415+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8c4145630 con 0x7ff8c40b4d60 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.415+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8b0001870 con 0x7ff8c40b4d60 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.415+0000 7ff8c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.415+0000 7ff8c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.415+0000 7ff8c17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:40.417 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.415+0000 7ff8c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.419+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 571848572 0 0) 0x7ff8a8004eb0 con 0x7ff8c414be90 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.419+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8c4141ff0 con 0x7ff8c414be90 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.419+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8bc001ea0 con 0x7ff8c414be90 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.419+0000 7ff8c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.419+0000 7ff8c17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.419+0000 7ff8c17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:40.421 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:40.422 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:40.422 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:40.422 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.435+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164057900 0 0) 0x7ff8c41422d0 con 0x7ff8c40b1730 2024-03-25T18:27:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.435+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8a8004eb0 con 0x7ff8c40b1730 2024-03-25T18:27:40.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.435+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8b8001910 con 0x7ff8c40b1730 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.435+0000 7ff8c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.435+0000 7ff8c17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.435+0000 7ff8c17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:40.437 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:40.438 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2825682056 0 0) 0x7ff8c4145630 con 0x7ff8c40b4d60 2024-03-25T18:27:40.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 >> v1:172.21.15.22:6790/0 conn(0x7ff8c40b1730 legacy=0x7ff8c4140480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 >> v1:172.21.15.22:6789/0 conn(0x7ff8c414be90 legacy=0x7ff8c41e3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:40.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:40.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 1 monclient: found mon.b 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c41452c0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 10 monclient: _renew_subs 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c4140fb0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 10 monclient: _renew_subs 2024-03-25T18:27:40.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:40.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c4140d70 con 0x7ff8c40b4d60 2024-03-25T18:27:40.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.436+0000 7ff8cabb5640 1 librados: init done 2024-03-25T18:27:40.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.437+0000 7ff8cabb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:40.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.437+0000 7ff8cabb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:40.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.437+0000 7ff8cabb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:40.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.437+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c41457c0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.735+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8b0003610 con 0x7ff8c40b4d60 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.735+0000 7ff8c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.735+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff8b0003920 con 0x7ff8c40b4d60 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.735+0000 7ff8c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.735+0000 7ff8c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.735+0000 7ff8c17fa640 10 monclient: dump: 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:40.737 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:40.738 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:40.738 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:40.738 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:40.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.735+0000 7ff8c2ffd640 4 set_mon_vals no callback set 2024-03-25T18:27:40.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.736+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7ff8b00198f0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.737+0000 7ff8c8951640 10 monclient: get_auth_request con 0x7ff8a806e0b0 auth_method 0 2024-03-25T18:27:40.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.737+0000 7ff8c17fa640 10 client.4606.objecter ms_handle_connect 0x7ff8a806e0b0 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7ff8b00535f0 con 0x7ff8c40b4d60 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 10 client.4606.objecter ms_dispatch 0x7ff8c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8b004ef30 con 0x7ff8c40b4d60 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:40.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.741+0000 7ff8c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:41.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.999+0000 7ff8cabb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:41.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.999+0000 7ff8cabb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:41.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.999+0000 7ff8cabb5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:41.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:40.999+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8c40627b0 con 0x7ff8c40b4d60 2024-03-25T18:27:41.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.028+0000 7ff8c17fa640 1 -- 172.21.15.22:0/2481551334 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7ff8b0051030 con 0x7ff8c40b4d60 2024-03-25T18:27:41.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.028+0000 7ff8c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:41.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.028+0000 7ff8c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:41.030 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:41.030 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:41.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 10 librados: watch_flush enter 2024-03-25T18:27:41.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 10 librados: watch_flush exit 2024-03-25T18:27:41.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:27:41.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:27:41.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 >> v1:172.21.15.33:6800/215350958 conn(0x7ff8a806e0b0 legacy=0x7ff8a8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 10 monclient: shutdown 2024-03-25T18:27:41.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:41.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 >> v1:172.21.15.33:6789/0 conn(0x7ff8c40b4d60 legacy=0x7ff8c4130fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 shutdown_connections 2024-03-25T18:27:41.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.030+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 >> 172.21.15.22:0/2481551334 conn(0x7ff8c40617a0 msgr2=0x7ff8c413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:41.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.031+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 shutdown_connections 2024-03-25T18:27:41.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.031+0000 7ff8cabb5640 1 -- 172.21.15.22:0/2481551334 wait complete. 2024-03-25T18:27:41.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:41.031+0000 7ff8cabb5640 1 librados: shutdown 2024-03-25T18:27:41.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:47.852 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:47.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.970+0000 7fce3c523640 10 monclient: get_monmap_and_config 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.970+0000 7fce3c523640 10 monclient: build_initial_monmap 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.970+0000 7fce3c523640 10 monclient: monmap: 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:47.970684+0000 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:47.970684+0000 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:47.971 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:47.972 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.970+0000 7fce3c523640 1 Processor -- start 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 1 -- start start 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient: init 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient(hunting): picked mon.noname-c con 0x7fce340a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient(hunting): picked mon.noname-b con 0x7fce3413f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:47.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient(hunting): picked mon.noname-a con 0x7fce341430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:47.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce34142520 con 0x7fce341430c0 2024-03-25T18:27:47.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340b28a0 con 0x7fce340a98f0 2024-03-25T18:27:47.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340b2a30 con 0x7fce3413f710 2024-03-25T18:27:47.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:47.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.971+0000 7fce3c523640 10 monclient(hunting): authenticate will time out at 7667.194441s 2024-03-25T18:27:47.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.987+0000 7fce39abe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fce3413f710 0x7fce341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41388/0 (socket says 172.21.15.22:41388) 2024-03-25T18:27:47.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.987+0000 7fce39abe640 1 -- 172.21.15.22:0/1757473802 learned_addr learned my addr 172.21.15.22:0/1757473802 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:47.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.997+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545088380 0 0) 0x7fce340b2a30 con 0x7fce3413f710 2024-03-25T18:27:47.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.997+0000 7fce392bd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:47.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.997+0000 7fce392bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:47.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.997+0000 7fce392bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:47.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.997+0000 7fce392bd640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:27:47.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:47.997+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce20003520 con 0x7fce3413f710 2024-03-25T18:27:48.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.002+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624593208 0 0) 0x7fce340b28a0 con 0x7fce340a98f0 2024-03-25T18:27:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.002+0000 7fce392bd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.002+0000 7fce392bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.002+0000 7fce392bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.002+0000 7fce392bd640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:27:48.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.002+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340b2a30 con 0x7fce340a98f0 2024-03-25T18:27:48.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.007+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77466581 0 0) 0x7fce34142520 con 0x7fce341430c0 2024-03-25T18:27:48.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.007+0000 7fce392bd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:48.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.007+0000 7fce392bd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:48.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.007+0000 7fce392bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:48.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.007+0000 7fce392bd640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:27:48.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.007+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340b28a0 con 0x7fce341430c0 2024-03-25T18:27:48.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2158992497 0 0) 0x7fce340b2a30 con 0x7fce340a98f0 2024-03-25T18:27:48.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 >> v1:172.21.15.22:6790/0 conn(0x7fce3413f710 legacy=0x7fce341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:48.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 >> v1:172.21.15.22:6789/0 conn(0x7fce341430c0 legacy=0x7fce3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:48.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:48.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 1 monclient: found mon.noname-c 2024-03-25T18:27:48.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340b44c0 con 0x7fce340a98f0 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 10 monclient: _finish_auth 0 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce3c523640 5 monclient: authenticate success, global_id 4609 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fce30001670 con 0x7fce340a98f0 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.016+0000 7fce392bd640 10 monclient: dump: 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.018 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.019 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:48.019 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:48.019 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:48.019 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:48.019 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:48.019 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:48.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.017+0000 7fce3c523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce300017d0 con 0x7fce340a98f0 2024-03-25T18:27:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce392bd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce392bd640 1 -- 172.21.15.22:0/1757473802 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fce30001be0 con 0x7fce340a98f0 2024-03-25T18:27:48.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce3c523640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce3c523640 4 set_mon_vals no callback set 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce392bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce392bd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.040+0000 7fce392bd640 10 monclient: dump: 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:48.041 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:48.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.041+0000 7fce3c523640 10 monclient: shutdown 2024-03-25T18:27:48.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.041+0000 7fce3c523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:48.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.041+0000 7fce3c523640 1 -- 172.21.15.22:0/1757473802 >> v1:172.21.15.33:6789/0 conn(0x7fce340a98f0 legacy=0x7fce340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:48.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.041+0000 7fce3c523640 1 -- 172.21.15.22:0/1757473802 shutdown_connections 2024-03-25T18:27:48.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.041+0000 7fce3c523640 1 -- 172.21.15.22:0/1757473802 >> 172.21.15.22:0/1757473802 conn(0x7fce340b7ea0 msgr2=0x7fce340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:48.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.041+0000 7fce3c523640 1 -- 172.21.15.22:0/1757473802 shutdown_connections 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.041+0000 7fce3c523640 1 -- 172.21.15.22:0/1757473802 wait complete. 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.042+0000 7fce3c523640 10 monclient: build_initial_monmap 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.042+0000 7fce3c523640 10 monclient: monmap: 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.042+0000 7fce3c523640 1 librados: starting msgr at 2024-03-25T18:27:48.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.042+0000 7fce3c523640 1 librados: starting objecter 2024-03-25T18:27:48.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.042+0000 7fce3c523640 1 Processor -- start 2024-03-25T18:27:48.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 1 -- start start 2024-03-25T18:27:48.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 1 librados: setting wanted keys 2024-03-25T18:27:48.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 1 librados: calling monclient init 2024-03-25T18:27:48.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient: init 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient(hunting): picked mon.noname-a con 0x7fce340a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient(hunting): picked mon.noname-b con 0x7fce3413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient(hunting): picked mon.noname-c con 0x7fce341430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce341390f0 con 0x7fce340a98f0 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce34138dd0 con 0x7fce3413f710 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce341346b0 con 0x7fce341430c0 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:48.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.043+0000 7fce3c523640 10 monclient(hunting): authenticate will time out at 7667.266492s 2024-03-25T18:27:48.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.047+0000 7fce3a2bf640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fce340a98f0 0x7fce34137e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:41904/0 (socket says 172.21.15.22:41904) 2024-03-25T18:27:48.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.047+0000 7fce3a2bf640 1 -- 172.21.15.22:0/255976309 learned_addr learned my addr 172.21.15.22:0/255976309 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:48.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.050+0000 7fce2affd640 10 client.?.objecter ms_handle_connect 0x7fce340a98f0 2024-03-25T18:27:48.051 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.050+0000 7fce2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:48.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.058+0000 7fce2affd640 10 client.?.objecter ms_handle_connect 0x7fce341430c0 2024-03-25T18:27:48.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.058+0000 7fce2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:48.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1683412185 0 0) 0x7fce341390f0 con 0x7fce340a98f0 2024-03-25T18:27:48.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:27:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18004eb0 con 0x7fce340a98f0 2024-03-25T18:27:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 10 client.?.objecter ms_handle_connect 0x7fce3413f710 2024-03-25T18:27:48.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.066+0000 7fce2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:48.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.073+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648770849 0 0) 0x7fce341346b0 con 0x7fce341430c0 2024-03-25T18:27:48.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.073+0000 7fce2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:48.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.073+0000 7fce2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:48.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.073+0000 7fce2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:48.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.073+0000 7fce2affd640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:27:48.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.073+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce341390f0 con 0x7fce341430c0 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.086+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2205741280 0 0) 0x7fce18004eb0 con 0x7fce340a98f0 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.086+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce341346b0 con 0x7fce340a98f0 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.086+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fce30001670 con 0x7fce340a98f0 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.086+0000 7fce2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.086+0000 7fce2affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.086+0000 7fce2affd640 10 monclient(hunting): dump: 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:48.087 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:48.088 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:48.088 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:48.088 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:48.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.086+0000 7fce2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:48.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.088+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192331577 0 0) 0x7fce34138dd0 con 0x7fce3413f710 2024-03-25T18:27:48.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.088+0000 7fce2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:48.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.088+0000 7fce2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:48.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.088+0000 7fce2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:48.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.088+0000 7fce2affd640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:27:48.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.088+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18004eb0 con 0x7fce3413f710 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.096+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1016816100 0 0) 0x7fce341390f0 con 0x7fce341430c0 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.096+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce34138dd0 con 0x7fce341430c0 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.096+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fce2c001b50 con 0x7fce341430c0 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.096+0000 7fce2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.096+0000 7fce2affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.096+0000 7fce2affd640 10 monclient(hunting): dump: 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:48.097 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:48.098 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:48.098 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:48.098 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:48.098 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.100+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970123424 0 0) 0x7fce18004eb0 con 0x7fce3413f710 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.101+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce341390f0 con 0x7fce3413f710 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.101+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fce240017b0 con 0x7fce3413f710 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.101+0000 7fce2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.101+0000 7fce2affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.101+0000 7fce2affd640 10 monclient(hunting): dump: 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.101 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.102 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:48.102 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:48.102 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:48.102 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:48.102 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:48.102 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2142805006 0 0) 0x7fce341390f0 con 0x7fce3413f710 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 >> v1:172.21.15.22:6790/0 conn(0x7fce341430c0 legacy=0x7fce34133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 >> v1:172.21.15.22:6789/0 conn(0x7fce340a98f0 legacy=0x7fce34137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 1 monclient: found mon.b 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34134840 con 0x7fce3413f710 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 10 monclient: _finish_auth 0 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:27:48.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 10 monclient: _renew_subs 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce34134a90 con 0x7fce3413f710 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 10 monclient: _renew_subs 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce341356f0 con 0x7fce3413f710 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.108+0000 7fce3c523640 1 librados: init done 2024-03-25T18:27:48.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.109+0000 7fce3c523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:48.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.109+0000 7fce3c523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:48.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.109+0000 7fce3c523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:48.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.109+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce340627b0 con 0x7fce3413f710 2024-03-25T18:27:48.395 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.395+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce24001cc0 con 0x7fce3413f710 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.395+0000 7fce2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.395+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fce24004980 con 0x7fce3413f710 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.395+0000 7fce2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.395+0000 7fce2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.395+0000 7fce2affd640 10 monclient: dump: 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:48.396 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:48.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.395+0000 7fce38abc640 4 set_mon_vals no callback set 2024-03-25T18:27:48.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.396+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fce2401a950 con 0x7fce3413f710 2024-03-25T18:27:48.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.396+0000 7fce3a2bf640 10 monclient: get_auth_request con 0x7fce1806e110 auth_method 0 2024-03-25T18:27:48.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.397+0000 7fce2affd640 10 client.4612.objecter ms_handle_connect 0x7fce1806e110 2024-03-25T18:27:48.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.402+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fce240546a0 con 0x7fce3413f710 2024-03-25T18:27:48.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.402+0000 7fce2affd640 10 client.4612.objecter ms_dispatch 0x7fce34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:48.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.402+0000 7fce2affd640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:48.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.402+0000 7fce2affd640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:48.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.402+0000 7fce2affd640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T18:27:48.403 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.403+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce2404ffe0 con 0x7fce3413f710 2024-03-25T18:27:48.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.403+0000 7fce2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:48.404 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.403+0000 7fce2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:48.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.645+0000 7fce3c523640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:48.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.645+0000 7fce3c523640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:48.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.645+0000 7fce3c523640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:48.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.645+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce341361b0 con 0x7fce3413f710 2024-03-25T18:27:48.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.674+0000 7fce2affd640 1 -- 172.21.15.22:0/255976309 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fce240520e0 con 0x7fce3413f710 2024-03-25T18:27:48.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.674+0000 7fce2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:48.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.674+0000 7fce2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:48.675 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:48.676 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:48.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 10 librados: watch_flush enter 2024-03-25T18:27:48.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 10 librados: watch_flush exit 2024-03-25T18:27:48.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T18:27:48.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 10 client.4612.objecter successfully canceled tick 2024-03-25T18:27:48.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 >> v1:172.21.15.33:6800/215350958 conn(0x7fce1806e110 legacy=0x7fce180705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:48.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 10 monclient: shutdown 2024-03-25T18:27:48.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:48.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 >> v1:172.21.15.33:6789/0 conn(0x7fce3413f710 legacy=0x7fce341386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:48.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 shutdown_connections 2024-03-25T18:27:48.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 >> 172.21.15.22:0/255976309 conn(0x7fce340b7ea0 msgr2=0x7fce34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:48.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 shutdown_connections 2024-03-25T18:27:48.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 1 -- 172.21.15.22:0/255976309 wait complete. 2024-03-25T18:27:48.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:48.677+0000 7fce3c523640 1 librados: shutdown 2024-03-25T18:27:48.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:55.497 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:55.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.613+0000 7ff944dc9640 10 monclient: get_monmap_and_config 2024-03-25T18:27:55.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.613+0000 7ff944dc9640 10 monclient: build_initial_monmap 2024-03-25T18:27:55.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.613+0000 7ff944dc9640 10 monclient: monmap: 2024-03-25T18:27:55.614 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:55.614 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:27:55.614076+0000 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:27:55.614076+0000 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:27:55.615 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.614+0000 7ff944dc9640 1 Processor -- start 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.614+0000 7ff944dc9640 1 -- start start 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.614+0000 7ff944dc9640 10 monclient: init 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 10 monclient(hunting): picked mon.noname-b con 0x7ff9400a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 10 monclient(hunting): picked mon.noname-a con 0x7ff94013f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 10 monclient(hunting): picked mon.noname-c con 0x7ff9401430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff940142520 con 0x7ff94013f710 2024-03-25T18:27:55.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9400b28a0 con 0x7ff9401430c0 2024-03-25T18:27:55.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9400b2a30 con 0x7ff9400a98f0 2024-03-25T18:27:55.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:55.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.615+0000 7ff944dc9640 10 monclient(hunting): authenticate will time out at 7674.838279s 2024-03-25T18:27:55.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.624+0000 7ff93ed9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7ff9401430c0 0x7ff94014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:35970/0 (socket says 172.21.15.22:35970) 2024-03-25T18:27:55.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.624+0000 7ff93ed9d640 1 -- 172.21.15.22:0/1691409587 learned_addr learned my addr 172.21.15.22:0/1691409587 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:55.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.631+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632099238 0 0) 0x7ff9400b28a0 con 0x7ff9401430c0 2024-03-25T18:27:55.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.631+0000 7ff93dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:27:55.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.631+0000 7ff93dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:55.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.631+0000 7ff93dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:55.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.631+0000 7ff93dd9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:27:55.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.631+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff92c003520 con 0x7ff9401430c0 2024-03-25T18:27:55.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2339418026 0 0) 0x7ff92c003520 con 0x7ff9401430c0 2024-03-25T18:27:55.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 >> v1:172.21.15.22:6790/0 conn(0x7ff9400a98f0 legacy=0x7ff9400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 >> v1:172.21.15.22:6789/0 conn(0x7ff94013f710 legacy=0x7ff9401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9400b44c0 con 0x7ff9401430c0 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff938001a10 con 0x7ff9401430c0 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff93dd9b640 10 monclient: dump: 2024-03-25T18:27:55.643 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff944dc9640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:27:55.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.642+0000 7ff944dc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:55.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff938001b70 con 0x7ff9401430c0 2024-03-25T18:27:55.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff93dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:55.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff93dd9b640 1 -- 172.21.15.22:0/1691409587 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff938001f80 con 0x7ff9401430c0 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff93dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff93dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff93dd9b640 10 monclient: dump: 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff944dc9640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:55.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.650+0000 7ff944dc9640 4 set_mon_vals no callback set 2024-03-25T18:27:55.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.651+0000 7ff944dc9640 10 monclient: shutdown 2024-03-25T18:27:55.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.651+0000 7ff944dc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:55.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.651+0000 7ff944dc9640 1 -- 172.21.15.22:0/1691409587 >> v1:172.21.15.33:6789/0 conn(0x7ff9401430c0 legacy=0x7ff94014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.651+0000 7ff944dc9640 1 -- 172.21.15.22:0/1691409587 shutdown_connections 2024-03-25T18:27:55.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.651+0000 7ff944dc9640 1 -- 172.21.15.22:0/1691409587 >> 172.21.15.22:0/1691409587 conn(0x7ff9400b7ea0 msgr2=0x7ff9400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:55.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.651+0000 7ff944dc9640 1 -- 172.21.15.22:0/1691409587 shutdown_connections 2024-03-25T18:27:55.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.651+0000 7ff944dc9640 1 -- 172.21.15.22:0/1691409587 wait complete. 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 10 monclient: build_initial_monmap 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 10 monclient: monmap: 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:55.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 1 librados: starting msgr at 2024-03-25T18:27:55.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 1 librados: starting objecter 2024-03-25T18:27:55.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 1 Processor -- start 2024-03-25T18:27:55.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 1 -- start start 2024-03-25T18:27:55.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 1 librados: setting wanted keys 2024-03-25T18:27:55.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 1 librados: calling monclient init 2024-03-25T18:27:55.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.652+0000 7ff944dc9640 10 monclient: init 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.653+0000 7ff944dc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.653+0000 7ff944dc9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.653+0000 7ff944dc9640 10 monclient(hunting): picked mon.noname-c con 0x7ff9400a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.653+0000 7ff944dc9640 10 monclient(hunting): picked mon.noname-a con 0x7ff94013f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.653+0000 7ff944dc9640 10 monclient(hunting): picked mon.noname-b con 0x7ff9401430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.653+0000 7ff944dc9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff940136790 con 0x7ff94013f710 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.653+0000 7ff944dc9640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9400abc30 con 0x7ff9401430c0 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.654+0000 7ff944dc9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9401d76c0 con 0x7ff9400a98f0 2024-03-25T18:27:55.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.654+0000 7ff944dc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:55.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.654+0000 7ff944dc9640 10 monclient(hunting): authenticate will time out at 7674.876615s 2024-03-25T18:27:55.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.661+0000 7ff93ed9d640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7ff9401430c0 0x7ff9401d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:35976/0 (socket says 172.21.15.22:35976) 2024-03-25T18:27:55.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.661+0000 7ff93ed9d640 1 -- 172.21.15.22:0/2003416740 learned_addr learned my addr 172.21.15.22:0/2003416740 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:27:55.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.663+0000 7ff9377fe640 10 client.?.objecter ms_handle_connect 0x7ff9401430c0 2024-03-25T18:27:55.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.663+0000 7ff9377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:55.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.667+0000 7ff9377fe640 10 client.?.objecter ms_handle_connect 0x7ff94013f710 2024-03-25T18:27:55.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.667+0000 7ff9377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:55.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.673+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410500381 0 0) 0x7ff9400abc30 con 0x7ff9401430c0 2024-03-25T18:27:55.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.673+0000 7ff9377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:55.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.673+0000 7ff9377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:55.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.673+0000 7ff9377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:55.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.673+0000 7ff9377fe640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:27:55.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.673+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff920004eb0 con 0x7ff9401430c0 2024-03-25T18:27:55.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.676+0000 7ff9377fe640 10 client.?.objecter ms_handle_connect 0x7ff9400a98f0 2024-03-25T18:27:55.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.676+0000 7ff9377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:55.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.683+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534886000 0 0) 0x7ff920004eb0 con 0x7ff9401430c0 2024-03-25T18:27:55.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.683+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9400abc30 con 0x7ff9401430c0 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.683+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff938001f40 con 0x7ff9401430c0 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.683+0000 7ff9377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.683+0000 7ff9377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.683+0000 7ff9377fe640 10 monclient(hunting): dump: 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:55.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.683+0000 7ff9377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.686+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341385069 0 0) 0x7ff9401d76c0 con 0x7ff9400a98f0 2024-03-25T18:27:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.686+0000 7ff9377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.686+0000 7ff9377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.686+0000 7ff9377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:55.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.686+0000 7ff9377fe640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:27:55.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.686+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff920004eb0 con 0x7ff9400a98f0 2024-03-25T18:27:55.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.690+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852224470 0 0) 0x7ff940136790 con 0x7ff94013f710 2024-03-25T18:27:55.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.691+0000 7ff9377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:55.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.691+0000 7ff9377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:55.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.691+0000 7ff9377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:55.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.691+0000 7ff9377fe640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:27:55.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.691+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9401d76c0 con 0x7ff94013f710 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2090426304 0 0) 0x7ff9400abc30 con 0x7ff9401430c0 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 >> v1:172.21.15.22:6790/0 conn(0x7ff9400a98f0 legacy=0x7ff940138720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 >> v1:172.21.15.22:6789/0 conn(0x7ff94013f710 legacy=0x7ff9401d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 1 monclient: found mon.b 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9401d41e0 con 0x7ff9401430c0 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff9377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:55.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 10 monclient: _renew_subs 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9401d82f0 con 0x7ff9401430c0 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 10 monclient: _renew_subs 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9401d50a0 con 0x7ff9401430c0 2024-03-25T18:27:55.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.693+0000 7ff944dc9640 1 librados: init done 2024-03-25T18:27:55.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.694+0000 7ff944dc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:55.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.694+0000 7ff944dc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:55.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.694+0000 7ff944dc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:55.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.694+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9401d43d0 con 0x7ff9401430c0 2024-03-25T18:27:55.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.701+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff938004e10 con 0x7ff9401430c0 2024-03-25T18:27:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.701+0000 7ff9377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:55.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.701+0000 7ff93d59a640 4 set_mon_vals no callback set 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.978+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff938005120 con 0x7ff9401430c0 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.978+0000 7ff9377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.978+0000 7ff9377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.978+0000 7ff9377fe640 10 monclient: dump: 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:27:55.979 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:27:55.980 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:27:55.980 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:27:55.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.979+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7ff93801b0f0 con 0x7ff9401430c0 2024-03-25T18:27:55.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.979+0000 7ff93e59c640 10 monclient: get_auth_request con 0x7ff92006de10 auth_method 0 2024-03-25T18:27:55.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.980+0000 7ff9377fe640 10 client.4618.objecter ms_handle_connect 0x7ff92006de10 2024-03-25T18:27:55.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.982+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7ff938054ce0 con 0x7ff9401430c0 2024-03-25T18:27:55.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.982+0000 7ff9377fe640 10 client.4618.objecter ms_dispatch 0x7ff940000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:27:55.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.982+0000 7ff9377fe640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:27:55.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.982+0000 7ff9377fe640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:27:55.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.982+0000 7ff9377fe640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T18:27:55.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.985+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9380507f0 con 0x7ff9401430c0 2024-03-25T18:27:55.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.985+0000 7ff9377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:55.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:55.985+0000 7ff9377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:56.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.228+0000 7ff944dc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:56.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.228+0000 7ff944dc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:56.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.228+0000 7ff944dc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:27:56.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.228+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9401d7850 con 0x7ff9401430c0 2024-03-25T18:27:56.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.246+0000 7ff9377fe640 1 -- 172.21.15.22:0/2003416740 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7ff938052720 con 0x7ff9401430c0 2024-03-25T18:27:56.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.246+0000 7ff9377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:56.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.246+0000 7ff9377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:56.246 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:27:56.246 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:56.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 10 librados: watch_flush enter 2024-03-25T18:27:56.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 10 librados: watch_flush exit 2024-03-25T18:27:56.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T18:27:56.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 10 client.4618.objecter successfully canceled tick 2024-03-25T18:27:56.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 >> v1:172.21.15.33:6800/215350958 conn(0x7ff92006de10 legacy=0x7ff9200702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:56.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 10 monclient: shutdown 2024-03-25T18:27:56.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:56.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 >> v1:172.21.15.33:6789/0 conn(0x7ff9401430c0 legacy=0x7ff9401d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:56.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 shutdown_connections 2024-03-25T18:27:56.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 >> 172.21.15.22:0/2003416740 conn(0x7ff9400b7ea0 msgr2=0x7ff940074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:56.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 shutdown_connections 2024-03-25T18:27:56.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 1 -- 172.21.15.22:0/2003416740 wait complete. 2024-03-25T18:27:56.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:27:56.248+0000 7ff944dc9640 1 librados: shutdown 2024-03-25T18:27:56.262 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:58.901 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:01.563 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.903 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:01.521 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.911 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:01.529 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.915 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:01.577 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.942 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:01.604 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.946 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:01.564 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.965 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:01.627 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.978 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:01.639 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:59.006 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:01.668 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:03.070 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:03.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.189+0000 7fb098617640 10 monclient: get_monmap_and_config 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.189+0000 7fb098617640 10 monclient: build_initial_monmap 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.189+0000 7fb098617640 10 monclient: monmap: 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:03.189521+0000 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:03.189521+0000 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:03.190 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:03.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.190+0000 7fb098617640 1 Processor -- start 2024-03-25T18:28:03.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.190+0000 7fb098617640 1 -- start start 2024-03-25T18:28:03.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.190+0000 7fb098617640 10 monclient: init 2024-03-25T18:28:03.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:03.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:03.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 10 monclient(hunting): picked mon.noname-c con 0x7fb0900a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:03.192 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 10 monclient(hunting): picked mon.noname-b con 0x7fb09013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:03.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 10 monclient(hunting): picked mon.noname-a con 0x7fb0901430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:03.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb090142520 con 0x7fb0901430c0 2024-03-25T18:28:03.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0900b28a0 con 0x7fb0900a98f0 2024-03-25T18:28:03.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0900b2a30 con 0x7fb09013f710 2024-03-25T18:28:03.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:03.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.191+0000 7fb098617640 10 monclient(hunting): authenticate will time out at 7682.414171s 2024-03-25T18:28:03.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.210+0000 7fb096bb4640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb0901430c0 0x7fb09014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39400/0 (socket says 172.21.15.22:39400) 2024-03-25T18:28:03.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.210+0000 7fb096bb4640 1 -- 172.21.15.22:0/441174216 learned_addr learned my addr 172.21.15.22:0/441174216 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:03.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.228+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315610364 0 0) 0x7fb0900b28a0 con 0x7fb0900a98f0 2024-03-25T18:28:03.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.228+0000 7fb0953b1640 10 monclient(hunting): _negotiate 2024-03-25T18:28:03.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.228+0000 7fb0953b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:03.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.228+0000 7fb0953b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:03.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.228+0000 7fb0953b1640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:28:03.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.228+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb07c003520 con 0x7fb0900a98f0 2024-03-25T18:28:03.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.230+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304517223 0 0) 0x7fb0900b2a30 con 0x7fb09013f710 2024-03-25T18:28:03.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.230+0000 7fb0953b1640 10 monclient(hunting): _negotiate 2024-03-25T18:28:03.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.230+0000 7fb0953b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:03.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.230+0000 7fb0953b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:03.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.230+0000 7fb0953b1640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:28:03.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.230+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0900b28a0 con 0x7fb09013f710 2024-03-25T18:28:03.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.233+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394001910 0 0) 0x7fb090142520 con 0x7fb0901430c0 2024-03-25T18:28:03.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.233+0000 7fb0953b1640 10 monclient(hunting): _negotiate 2024-03-25T18:28:03.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.233+0000 7fb0953b1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:03.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.233+0000 7fb0953b1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:03.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.233+0000 7fb0953b1640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:28:03.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.233+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0900b2a30 con 0x7fb0901430c0 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 738670521 0 0) 0x7fb07c003520 con 0x7fb0900a98f0 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 >> v1:172.21.15.22:6790/0 conn(0x7fb09013f710 legacy=0x7fb0901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 >> v1:172.21.15.22:6789/0 conn(0x7fb0901430c0 legacy=0x7fb09014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 10 monclient: _finish_hunting 0 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 1 monclient: found mon.noname-c 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0900b44c0 con 0x7fb0900a98f0 2024-03-25T18:28:03.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 10 monclient: _finish_auth 0 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 10 monclient: _check_auth_tickets 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb08c001690 con 0x7fb0900a98f0 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb0953b1640 10 monclient: dump: 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:03.242 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:03.243 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:03.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb098617640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:28:03.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.240+0000 7fb098617640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb08c0017f0 con 0x7fb0900a98f0 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb0953b1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb0953b1640 1 -- 172.21.15.22:0/441174216 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb08c001c00 con 0x7fb0900a98f0 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb098617640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb098617640 4 set_mon_vals no callback set 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb0953b1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb0953b1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.279+0000 7fb0953b1640 10 monclient: dump: 2024-03-25T18:28:03.281 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.280+0000 7fb098617640 10 monclient: shutdown 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.280+0000 7fb098617640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.280+0000 7fb098617640 1 -- 172.21.15.22:0/441174216 >> v1:172.21.15.33:6789/0 conn(0x7fb0900a98f0 legacy=0x7fb0900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.280+0000 7fb098617640 1 -- 172.21.15.22:0/441174216 shutdown_connections 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.280+0000 7fb098617640 1 -- 172.21.15.22:0/441174216 >> 172.21.15.22:0/441174216 conn(0x7fb0900b7ea0 msgr2=0x7fb0900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:03.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.280+0000 7fb098617640 1 -- 172.21.15.22:0/441174216 shutdown_connections 2024-03-25T18:28:03.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.280+0000 7fb098617640 1 -- 172.21.15.22:0/441174216 wait complete. 2024-03-25T18:28:03.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.281+0000 7fb098617640 10 monclient: build_initial_monmap 2024-03-25T18:28:03.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.281+0000 7fb098617640 10 monclient: monmap: 2024-03-25T18:28:03.284 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:03.284 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:03.284 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:03.284 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:03.284 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 1 librados: starting msgr at 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 1 librados: starting objecter 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 1 Processor -- start 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 1 -- start start 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 1 librados: setting wanted keys 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 1 librados: calling monclient init 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 10 monclient: init 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 10 monclient(hunting): picked mon.noname-c con 0x7fb0900a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:03.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 10 monclient(hunting): picked mon.noname-a con 0x7fb09013f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:03.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 10 monclient(hunting): picked mon.noname-b con 0x7fb0901430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:03.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.282+0000 7fb098617640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb090139870 con 0x7fb09013f710 2024-03-25T18:28:03.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.283+0000 7fb098617640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0901d51f0 con 0x7fb0901430c0 2024-03-25T18:28:03.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.283+0000 7fb098617640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb090138830 con 0x7fb0900a98f0 2024-03-25T18:28:03.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.283+0000 7fb098617640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:03.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.283+0000 7fb098617640 10 monclient(hunting): authenticate will time out at 7682.505616s 2024-03-25T18:28:03.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.302+0000 7fb095bb2640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb09013f710 0x7fb0901d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39436/0 (socket says 172.21.15.22:39436) 2024-03-25T18:28:03.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.302+0000 7fb095bb2640 1 -- 172.21.15.22:0/99528726 learned_addr learned my addr 172.21.15.22:0/99528726 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:03.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.305+0000 7fb086ffd640 10 client.?.objecter ms_handle_connect 0x7fb09013f710 2024-03-25T18:28:03.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.305+0000 7fb086ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:03.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.305+0000 7fb086ffd640 10 client.?.objecter ms_handle_connect 0x7fb0900a98f0 2024-03-25T18:28:03.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.305+0000 7fb086ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:03.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.319+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57386798 0 0) 0x7fb090139870 con 0x7fb09013f710 2024-03-25T18:28:03.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.319+0000 7fb086ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:03.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.319+0000 7fb086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:03.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.319+0000 7fb086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:03.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.319+0000 7fb086ffd640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:28:03.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.319+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb074004eb0 con 0x7fb09013f710 2024-03-25T18:28:03.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.320+0000 7fb086ffd640 10 client.?.objecter ms_handle_connect 0x7fb0901430c0 2024-03-25T18:28:03.321 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.320+0000 7fb086ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:03.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.323+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779275500 0 0) 0x7fb090138830 con 0x7fb0900a98f0 2024-03-25T18:28:03.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.323+0000 7fb086ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:03.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.323+0000 7fb086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:03.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.323+0000 7fb086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:03.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.323+0000 7fb086ffd640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:28:03.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.323+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb090139870 con 0x7fb0900a98f0 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.336+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1366355939 0 0) 0x7fb074004eb0 con 0x7fb09013f710 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.336+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb090138830 con 0x7fb09013f710 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.336+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb080001a30 con 0x7fb09013f710 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.336+0000 7fb086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.336+0000 7fb086ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.336+0000 7fb086ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.337 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.338 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:03.338 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:03.338 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:03.338 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:03.338 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:03.338 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:03.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.336+0000 7fb086ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:03.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2275144253 0 0) 0x7fb090139870 con 0x7fb0900a98f0 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb074004eb0 con 0x7fb0900a98f0 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb08c001b00 con 0x7fb0900a98f0 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 10 monclient(hunting): dump: 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:03.342 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:03.343 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:03.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735638264 0 0) 0x7fb0901d51f0 con 0x7fb0901430c0 2024-03-25T18:28:03.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:03.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:03.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:03.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:28:03.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.341+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb090139870 con 0x7fb0901430c0 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2334531924 0 0) 0x7fb074004eb0 con 0x7fb0900a98f0 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 >> v1:172.21.15.33:6789/0 conn(0x7fb0901430c0 legacy=0x7fb0901e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 >> v1:172.21.15.22:6789/0 conn(0x7fb09013f710 legacy=0x7fb0901d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 1 monclient: found mon.c 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0901d77a0 con 0x7fb0900a98f0 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 5 monclient: authenticate success, global_id 4628 2024-03-25T18:28:03.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 10 monclient: _renew_subs 2024-03-25T18:28:03.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:03.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0901d83d0 con 0x7fb0900a98f0 2024-03-25T18:28:03.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:03.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 10 monclient: _renew_subs 2024-03-25T18:28:03.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:03.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb090138c80 con 0x7fb0900a98f0 2024-03-25T18:28:03.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.348+0000 7fb098617640 1 librados: init done 2024-03-25T18:28:03.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.349+0000 7fb098617640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:03.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.349+0000 7fb098617640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:03.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.349+0000 7fb098617640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:03.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.349+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0900b3e80 con 0x7fb0900a98f0 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.699+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb08c001ca0 con 0x7fb0900a98f0 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.699+0000 7fb086ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.699+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fb08c003a40 con 0x7fb0900a98f0 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.699+0000 7fb086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.699+0000 7fb086ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.699+0000 7fb086ffd640 10 monclient: dump: 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:03.700 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:03.701 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:03.701 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:03.701 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:03.701 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:03.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.699+0000 7fb094bb0640 4 set_mon_vals no callback set 2024-03-25T18:28:03.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.700+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fb08c019a10 con 0x7fb0900a98f0 2024-03-25T18:28:03.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.701+0000 7fb095bb2640 10 monclient: get_auth_request con 0x7fb07406df50 auth_method 0 2024-03-25T18:28:03.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.701+0000 7fb086ffd640 10 client.4628.objecter ms_handle_connect 0x7fb07406df50 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fb08c053620 con 0x7fb0900a98f0 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 10 client.4628.objecter ms_dispatch 0x7fb090000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb08c04f130 con 0x7fb0900a98f0 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:03.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.706+0000 7fb086ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:03.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.928+0000 7fb098617640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:03.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.928+0000 7fb098617640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:03.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.928+0000 7fb098617640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:03.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.928+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb090074cc0 con 0x7fb0900a98f0 2024-03-25T18:28:03.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.959+0000 7fb086ffd640 1 -- 172.21.15.22:0/99528726 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fb08c051060 con 0x7fb0900a98f0 2024-03-25T18:28:03.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.959+0000 7fb086ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:03.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.959+0000 7fb086ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:03.960 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:03.960 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:03.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 10 librados: watch_flush enter 2024-03-25T18:28:03.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 10 librados: watch_flush exit 2024-03-25T18:28:03.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-25T18:28:03.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 10 client.4628.objecter successfully canceled tick 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 >> v1:172.21.15.33:6800/215350958 conn(0x7fb07406df50 legacy=0x7fb074070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 10 monclient: shutdown 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 >> v1:172.21.15.22:6790/0 conn(0x7fb0900a98f0 legacy=0x7fb090138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 shutdown_connections 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 >> 172.21.15.22:0/99528726 conn(0x7fb0900b7ea0 msgr2=0x7fb090074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 shutdown_connections 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 1 -- 172.21.15.22:0/99528726 wait complete. 2024-03-25T18:28:03.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:03.961+0000 7fb098617640 1 librados: shutdown 2024-03-25T18:28:03.975 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:10.782 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:10.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.897+0000 7f5413d04640 10 monclient: get_monmap_and_config 2024-03-25T18:28:10.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.897+0000 7f5413d04640 10 monclient: build_initial_monmap 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.897+0000 7f5413d04640 10 monclient: monmap: 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:10.897586+0000 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:10.897586+0000 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:10.898 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.897+0000 7f5413d04640 1 Processor -- start 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 1 -- start start 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient: init 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient(hunting): picked mon.noname-c con 0x7f540c0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient(hunting): picked mon.noname-a con 0x7f540c0b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient(hunting): picked mon.noname-b con 0x7f540c14be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c0af940 con 0x7f540c0b1730 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c0b0a70 con 0x7f540c0b4d60 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c0b0d50 con 0x7f540c14be90 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:10.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.898+0000 7f5413d04640 10 monclient(hunting): authenticate will time out at 7690.121308s 2024-03-25T18:28:10.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.899+0000 7f54122a1640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f540c14be90 0x7f540c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60210/0 (socket says 172.21.15.22:60210) 2024-03-25T18:28:10.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.899+0000 7f54122a1640 1 -- 172.21.15.22:0/3699318445 learned_addr learned my addr 172.21.15.22:0/3699318445 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:10.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.975+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871317446 0 0) 0x7f540c0af940 con 0x7f540c0b1730 2024-03-25T18:28:10.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.975+0000 7f5410a9e640 10 monclient(hunting): _negotiate 2024-03-25T18:28:10.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.975+0000 7f5410a9e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:10.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.975+0000 7f5410a9e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:10.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.975+0000 7f5410a9e640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:28:10.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:10.975+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f8003520 con 0x7f540c0b1730 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1162032757 0 0) 0x7f53f8003520 con 0x7f540c0b1730 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 >> v1:172.21.15.22:6790/0 conn(0x7f540c14be90 legacy=0x7f540c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 >> v1:172.21.15.33:6789/0 conn(0x7f540c0b4d60 legacy=0x7f540c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 10 monclient: _finish_hunting 0 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 1 monclient: found mon.noname-a 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c0b1030 con 0x7f540c0b1730 2024-03-25T18:28:11.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 10 monclient: _finish_auth 0 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 10 monclient: _check_auth_tickets 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53fc001690 con 0x7f540c0b1730 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5413d04640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5413d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5410a9e640 10 monclient: dump: 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.013 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:11.014 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:11.014 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:11.014 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:11.014 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:11.014 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:11.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.011+0000 7f5413d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:11.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53fc0017f0 con 0x7f540c0b1730 2024-03-25T18:28:11.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5410a9e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:11.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5410a9e640 1 -- 172.21.15.22:0/3699318445 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f53fc001c00 con 0x7f540c0b1730 2024-03-25T18:28:11.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5410a9e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:11.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5410a9e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:11.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5410a9e640 10 monclient: dump: 2024-03-25T18:28:11.026 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:11.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5413d04640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:11.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.025+0000 7f5413d04640 4 set_mon_vals no callback set 2024-03-25T18:28:11.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.026+0000 7f5413d04640 10 monclient: shutdown 2024-03-25T18:28:11.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.026+0000 7f5413d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:11.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.026+0000 7f5413d04640 1 -- 172.21.15.22:0/3699318445 >> v1:172.21.15.22:6789/0 conn(0x7f540c0b1730 legacy=0x7f540c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.026+0000 7f5413d04640 1 -- 172.21.15.22:0/3699318445 shutdown_connections 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.026+0000 7f5413d04640 1 -- 172.21.15.22:0/3699318445 >> 172.21.15.22:0/3699318445 conn(0x7f540c0617a0 msgr2=0x7f540c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.026+0000 7f5413d04640 1 -- 172.21.15.22:0/3699318445 shutdown_connections 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.026+0000 7f5413d04640 1 -- 172.21.15.22:0/3699318445 wait complete. 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.027+0000 7f5413d04640 10 monclient: build_initial_monmap 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.027+0000 7f5413d04640 10 monclient: monmap: 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:11.031 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:11.032 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:11.032 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:11.032 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:11.032 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:11.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 1 librados: starting msgr at 2024-03-25T18:28:11.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 1 librados: starting objecter 2024-03-25T18:28:11.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 1 Processor -- start 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 1 -- start start 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 1 librados: setting wanted keys 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 1 librados: calling monclient init 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 10 monclient: init 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.028+0000 7f5413d04640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 10 monclient(hunting): picked mon.noname-a con 0x7f540c0b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:11.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 10 monclient(hunting): picked mon.noname-b con 0x7f540c0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:11.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 10 monclient(hunting): picked mon.noname-c con 0x7f540c14be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:11.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c1d7940 con 0x7f540c0b1730 2024-03-25T18:28:11.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c1d8570 con 0x7f540c0b4d60 2024-03-25T18:28:11.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f540c1d8700 con 0x7f540c14be90 2024-03-25T18:28:11.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:11.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.029+0000 7f5413d04640 10 monclient(hunting): authenticate will time out at 7690.251819s 2024-03-25T18:28:11.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.033+0000 7f541129f640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f540c0b4d60 0x7f540c136e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:53044/0 (socket says 172.21.15.22:53044) 2024-03-25T18:28:11.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.033+0000 7f541129f640 1 -- 172.21.15.22:0/482315725 learned_addr learned my addr 172.21.15.22:0/482315725 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:11.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.034+0000 7f54027fc640 10 client.?.objecter ms_handle_connect 0x7f540c0b4d60 2024-03-25T18:28:11.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.034+0000 7f54027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:11.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.034+0000 7f54027fc640 10 client.?.objecter ms_handle_connect 0x7f540c14be90 2024-03-25T18:28:11.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.034+0000 7f54027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:11.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245690610 0 0) 0x7f540c1d8570 con 0x7f540c0b4d60 2024-03-25T18:28:11.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:11.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:11.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:11.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:28:11.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53f0004eb0 con 0x7f540c0b4d60 2024-03-25T18:28:11.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551243737 0 0) 0x7f540c1d8700 con 0x7f540c14be90 2024-03-25T18:28:11.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:11.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:11.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:11.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:28:11.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.041+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c1d8570 con 0x7f540c14be90 2024-03-25T18:28:11.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.043+0000 7f54027fc640 10 client.?.objecter ms_handle_connect 0x7f540c0b1730 2024-03-25T18:28:11.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.043+0000 7f54027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.052+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866098178 0 0) 0x7f540c1d8570 con 0x7f540c14be90 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.052+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f540c1d8700 con 0x7f540c14be90 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.052+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5404001b00 con 0x7f540c14be90 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.052+0000 7f54027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.052+0000 7f54027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.052+0000 7f54027fc640 10 monclient(hunting): dump: 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.053 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.054 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:11.054 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:11.054 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:11.054 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:11.054 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:11.054 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:11.054 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.052+0000 7f54027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:11.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.060+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511503302 0 0) 0x7f540c1d7940 con 0x7f540c0b1730 2024-03-25T18:28:11.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.060+0000 7f54027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:11.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.060+0000 7f54027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:11.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.060+0000 7f54027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:11.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.060+0000 7f54027fc640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:28:11.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.060+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f540c1d8570 con 0x7f540c0b1730 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 531471016 0 0) 0x7f540c1d8700 con 0x7f540c14be90 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 >> v1:172.21.15.33:6789/0 conn(0x7f540c0b4d60 legacy=0x7f540c136e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 >> v1:172.21.15.22:6789/0 conn(0x7f540c0b1730 legacy=0x7f540c1d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 1 monclient: found mon.c 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f540c1361c0 con 0x7f540c14be90 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f54027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 5 monclient: authenticate success, global_id 4634 2024-03-25T18:28:11.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 10 monclient: _renew_subs 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540c136470 con 0x7f540c14be90 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 10 monclient: _renew_subs 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f540c137790 con 0x7f540c14be90 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.064+0000 7f5413d04640 1 librados: init done 2024-03-25T18:28:11.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.065+0000 7f5413d04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:11.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.065+0000 7f5413d04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:11.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.065+0000 7f5413d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:11.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.065+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f540c0627b0 con 0x7f540c14be90 2024-03-25T18:28:11.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.376+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5404001f30 con 0x7f540c14be90 2024-03-25T18:28:11.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.376+0000 7f54027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:11.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.376+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5404004d00 con 0x7f540c14be90 2024-03-25T18:28:11.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.376+0000 7f54027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:11.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.376+0000 7f54027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.376+0000 7f54027fc640 10 monclient: dump: 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.376+0000 7f5403fff640 4 set_mon_vals no callback set 2024-03-25T18:28:11.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.377+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f5404004f20 con 0x7f540c14be90 2024-03-25T18:28:11.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.377+0000 7f5411aa0640 10 monclient: get_auth_request con 0x7f53f006de10 auth_method 0 2024-03-25T18:28:11.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.378+0000 7f54027fc640 10 client.4634.objecter ms_handle_connect 0x7f53f006de10 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f54040548d0 con 0x7f540c14be90 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 10 client.4634.objecter ms_dispatch 0x7f540c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54040503e0 con 0x7f540c14be90 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:11.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.382+0000 7f54027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:11.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.633+0000 7f5413d04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:11.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.633+0000 7f5413d04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:11.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.633+0000 7f5413d04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:11.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.633+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f540c069f40 con 0x7f540c14be90 2024-03-25T18:28:11.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.664+0000 7f54027fc640 1 -- 172.21.15.22:0/482315725 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f5404052310 con 0x7f540c14be90 2024-03-25T18:28:11.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.664+0000 7f54027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:11.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.664+0000 7f54027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:11.666 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:11.667 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:11.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.666+0000 7f5413d04640 10 librados: watch_flush enter 2024-03-25T18:28:11.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.666+0000 7f5413d04640 10 librados: watch_flush exit 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.666+0000 7f5413d04640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.666+0000 7f5413d04640 10 client.4634.objecter successfully canceled tick 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.666+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 >> v1:172.21.15.33:6800/215350958 conn(0x7f53f006de10 legacy=0x7f53f00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 10 monclient: shutdown 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 >> v1:172.21.15.22:6790/0 conn(0x7f540c14be90 legacy=0x7f540c12fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 shutdown_connections 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 >> 172.21.15.22:0/482315725 conn(0x7f540c0617a0 msgr2=0x7f540c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 shutdown_connections 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 1 -- 172.21.15.22:0/482315725 wait complete. 2024-03-25T18:28:11.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:11.667+0000 7f5413d04640 1 librados: shutdown 2024-03-25T18:28:11.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:18.487 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:18.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.604+0000 7f49c51f4640 10 monclient: get_monmap_and_config 2024-03-25T18:28:18.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.604+0000 7f49c51f4640 10 monclient: build_initial_monmap 2024-03-25T18:28:18.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.604+0000 7f49c51f4640 10 monclient: monmap: 2024-03-25T18:28:18.605 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:18.605 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:18.605347+0000 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:18.605347+0000 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:18.606 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:18.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.604+0000 7f49c51f4640 1 Processor -- start 2024-03-25T18:28:18.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.605+0000 7f49c51f4640 1 -- start start 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.605+0000 7f49c51f4640 10 monclient: init 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.605+0000 7f49c51f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.605+0000 7f49c51f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.605+0000 7f49c51f4640 10 monclient(hunting): picked mon.noname-c con 0x7f49c006c380 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.606+0000 7f49c51f4640 10 monclient(hunting): picked mon.noname-b con 0x7f49c00b12d0 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.606+0000 7f49c51f4640 10 monclient(hunting): picked mon.noname-a con 0x7f49c00aedf0 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.606+0000 7f49c51f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00b16f0 con 0x7f49c00aedf0 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.606+0000 7f49c51f4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00adfe0 con 0x7f49c006c380 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.606+0000 7f49c51f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00add00 con 0x7f49c00b12d0 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.606+0000 7f49c51f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:18.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.606+0000 7f49c51f4640 10 monclient(hunting): authenticate will time out at 7697.829676s 2024-03-25T18:28:18.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.612+0000 7f49bf59e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f49c00aedf0 0x7f49c014e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52836/0 (socket says 172.21.15.22:52836) 2024-03-25T18:28:18.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.612+0000 7f49bf59e640 1 -- 172.21.15.22:0/1974290945 learned_addr learned my addr 172.21.15.22:0/1974290945 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:18.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.630+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287152009 0 0) 0x7f49c00adfe0 con 0x7f49c006c380 2024-03-25T18:28:18.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.630+0000 7f49be59c640 10 monclient(hunting): _negotiate 2024-03-25T18:28:18.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.630+0000 7f49be59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:18.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.630+0000 7f49be59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:18.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.630+0000 7f49be59c640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:28:18.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.630+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49ac003520 con 0x7f49c006c380 2024-03-25T18:28:18.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.634+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389183745 0 0) 0x7f49c00b16f0 con 0x7f49c00aedf0 2024-03-25T18:28:18.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.634+0000 7f49be59c640 10 monclient(hunting): _negotiate 2024-03-25T18:28:18.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.634+0000 7f49be59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:18.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.634+0000 7f49be59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:18.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.634+0000 7f49be59c640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:28:18.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.634+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c00adfe0 con 0x7f49c00aedf0 2024-03-25T18:28:18.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1869192349 0 0) 0x7f49ac003520 con 0x7f49c006c380 2024-03-25T18:28:18.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 >> v1:172.21.15.22:6790/0 conn(0x7f49c00b12d0 legacy=0x7f49c00ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 >> v1:172.21.15.22:6789/0 conn(0x7f49c00aedf0 legacy=0x7f49c014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 10 monclient: _finish_hunting 0 2024-03-25T18:28:18.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 1 monclient: found mon.noname-c 2024-03-25T18:28:18.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:18.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00ae2c0 con 0x7f49c006c380 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 10 monclient: _finish_auth 0 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 10 monclient: _check_auth_tickets 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f49b0001690 con 0x7f49c006c380 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49be59c640 10 monclient: dump: 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:18.650 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:18.651 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:18.651 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:18.651 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:18.651 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:18.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49c51f4640 5 monclient: authenticate success, global_id 4633 2024-03-25T18:28:18.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.647+0000 7f49c51f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49b00017f0 con 0x7f49c006c380 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49be59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49be59c640 1 -- 172.21.15.22:0/1974290945 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f49b0001b80 con 0x7f49c006c380 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49c51f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49c51f4640 4 set_mon_vals no callback set 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49be59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49be59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:18.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.657+0000 7f49be59c640 10 monclient: dump: 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:18.660 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:18.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.658+0000 7f49c51f4640 10 monclient: shutdown 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.658+0000 7f49c51f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.658+0000 7f49c51f4640 1 -- 172.21.15.22:0/1974290945 >> v1:172.21.15.33:6789/0 conn(0x7f49c006c380 legacy=0x7f49c00b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.658+0000 7f49c51f4640 1 -- 172.21.15.22:0/1974290945 shutdown_connections 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.658+0000 7f49c51f4640 1 -- 172.21.15.22:0/1974290945 >> 172.21.15.22:0/1974290945 conn(0x7f49c00b5d10 msgr2=0x7f49c00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.658+0000 7f49c51f4640 1 -- 172.21.15.22:0/1974290945 shutdown_connections 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.658+0000 7f49c51f4640 1 -- 172.21.15.22:0/1974290945 wait complete. 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.659+0000 7f49c51f4640 10 monclient: build_initial_monmap 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.659+0000 7f49c51f4640 10 monclient: monmap: 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:18.662 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.660+0000 7f49c51f4640 1 librados: starting msgr at 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.660+0000 7f49c51f4640 1 librados: starting objecter 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.660+0000 7f49c51f4640 1 Processor -- start 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.660+0000 7f49c51f4640 1 -- start start 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.660+0000 7f49c51f4640 1 librados: setting wanted keys 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.660+0000 7f49c51f4640 1 librados: calling monclient init 2024-03-25T18:28:18.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.660+0000 7f49c51f4640 10 monclient: init 2024-03-25T18:28:18.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:18.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:28:18.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 10 monclient(hunting): picked mon.noname-c con 0x7f49c006c380 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:18.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 10 monclient(hunting): picked mon.noname-b con 0x7f49c00aedf0 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:18.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 10 monclient(hunting): picked mon.noname-a con 0x7f49c00b12d0 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:18.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c006c8e0 con 0x7f49c00b12d0 2024-03-25T18:28:18.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c01c4960 con 0x7f49c00aedf0 2024-03-25T18:28:18.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c01db480 con 0x7f49c006c380 2024-03-25T18:28:18.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:18.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.661+0000 7f49c51f4640 10 monclient(hunting): authenticate will time out at 7697.884952s 2024-03-25T18:28:18.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.667+0000 7f49bf59e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f49c00b12d0 0x7f49c01c2300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52896/0 (socket says 172.21.15.22:52896) 2024-03-25T18:28:18.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.667+0000 7f49bf59e640 1 -- 172.21.15.22:0/198107472 learned_addr learned my addr 172.21.15.22:0/198107472 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:18.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.669+0000 7f49b7fff640 10 client.?.objecter ms_handle_connect 0x7f49c00b12d0 2024-03-25T18:28:18.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.669+0000 7f49b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.674+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227868435 0 0) 0x7f49c006c8e0 con 0x7f49c00b12d0 2024-03-25T18:28:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.674+0000 7f49b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.674+0000 7f49b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.674+0000 7f49b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.674+0000 7f49b7fff640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:28:18.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.674+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49a0004eb0 con 0x7f49c00b12d0 2024-03-25T18:28:18.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.677+0000 7f49b7fff640 10 client.?.objecter ms_handle_connect 0x7f49c00aedf0 2024-03-25T18:28:18.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.677+0000 7f49b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:18.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.691+0000 7f49b7fff640 10 client.?.objecter ms_handle_connect 0x7f49c006c380 2024-03-25T18:28:18.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.691+0000 7f49b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:18.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.692+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979746214 0 0) 0x7f49c01c4960 con 0x7f49c00aedf0 2024-03-25T18:28:18.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.692+0000 7f49b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:18.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.692+0000 7f49b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:18.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.692+0000 7f49b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:18.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.692+0000 7f49b7fff640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:28:18.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.692+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c006c8e0 con 0x7f49c00aedf0 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.702+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3513316722 0 0) 0x7f49a0004eb0 con 0x7f49c00b12d0 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.702+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49c01c4960 con 0x7f49c00b12d0 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.702+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f49b8001cb0 con 0x7f49c00b12d0 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.702+0000 7f49b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.702+0000 7f49b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.702+0000 7f49b7fff640 10 monclient(hunting): dump: 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:18.704 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:18.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.702+0000 7f49b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.704+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2050657503 0 0) 0x7f49c006c8e0 con 0x7f49c00aedf0 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.704+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49a0004eb0 con 0x7f49c00aedf0 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.704+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f49a8001800 con 0x7f49c00aedf0 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.704+0000 7f49b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.704+0000 7f49b7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.704+0000 7f49b7fff640 10 monclient(hunting): dump: 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:18.706 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.706+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596591729 0 0) 0x7f49c01db480 con 0x7f49c006c380 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.706+0000 7f49b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.706+0000 7f49b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.706+0000 7f49b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:18.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.706+0000 7f49b7fff640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:28:18.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.706+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49c006c8e0 con 0x7f49c006c380 2024-03-25T18:28:18.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3132623242 0 0) 0x7f49a0004eb0 con 0x7f49c00aedf0 2024-03-25T18:28:18.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 >> v1:172.21.15.22:6790/0 conn(0x7f49c006c380 legacy=0x7f49c01dacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 >> v1:172.21.15.22:6789/0 conn(0x7f49c00b12d0 legacy=0x7f49c01c2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 1 monclient: found mon.b 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c01c1890 con 0x7f49c00aedf0 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 10 monclient: _renew_subs 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c01c0ef0 con 0x7f49c00aedf0 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:18.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 10 monclient: _renew_subs 2024-03-25T18:28:18.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:18.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c01c2bf0 con 0x7f49c00aedf0 2024-03-25T18:28:18.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.710+0000 7f49c51f4640 1 librados: init done 2024-03-25T18:28:18.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.711+0000 7f49c51f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:18.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.711+0000 7f49c51f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:18.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.711+0000 7f49c51f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:18.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.711+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c01c2ea0 con 0x7f49c00aedf0 2024-03-25T18:28:18.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.986+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49a8001cb0 con 0x7f49c00aedf0 2024-03-25T18:28:18.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.986+0000 7f49b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:18.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.987+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f49a8003790 con 0x7f49c00aedf0 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.987+0000 7f49bdd9b640 4 set_mon_vals no callback set 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.987+0000 7f49b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.987+0000 7f49b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.987+0000 7f49b7fff640 10 monclient: dump: 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:18.989 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:18.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.988+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f49a8003970 con 0x7f49c00aedf0 2024-03-25T18:28:18.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.989+0000 7f49bed9d640 10 monclient: get_auth_request con 0x7f49a006dfe0 auth_method 0 2024-03-25T18:28:18.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.989+0000 7f49b7fff640 10 client.4636.objecter ms_handle_connect 0x7f49a006dfe0 2024-03-25T18:28:18.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.993+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f49a8053560 con 0x7f49c00aedf0 2024-03-25T18:28:18.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.993+0000 7f49b7fff640 10 client.4636.objecter ms_dispatch 0x7f49c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:18.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.993+0000 7f49b7fff640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:18.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.993+0000 7f49b7fff640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:18.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.993+0000 7f49b7fff640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T18:28:18.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.994+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49a804eea0 con 0x7f49c00aedf0 2024-03-25T18:28:18.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.994+0000 7f49b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:18.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:18.994+0000 7f49b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:19.059 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:21.721 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.065 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:21.727 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.106 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:21.767 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.144 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:21.762 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.154 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:21.772 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.167 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:21.828 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.177 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:21.839 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.184 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:21.802 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.200 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:21.862 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:19.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.238+0000 7f49c51f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:19.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.238+0000 7f49c51f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:19.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.238+0000 7f49c51f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:19.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.238+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49c01c11f0 con 0x7f49c00aedf0 2024-03-25T18:28:19.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.271+0000 7f49b7fff640 1 -- 172.21.15.22:0/198107472 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f49c01c11f0 con 0x7f49c00aedf0 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.271+0000 7f49b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.271+0000 7f49b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:19.273 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 10 librados: watch_flush enter 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 10 librados: watch_flush exit 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 10 client.4636.objecter successfully canceled tick 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 >> v1:172.21.15.33:6800/215350958 conn(0x7f49a006dfe0 legacy=0x7f49a00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 10 monclient: shutdown 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 >> v1:172.21.15.33:6789/0 conn(0x7f49c00aedf0 legacy=0x7f49c01c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 shutdown_connections 2024-03-25T18:28:19.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 >> 172.21.15.22:0/198107472 conn(0x7f49c00b5d10 msgr2=0x7f49c00b5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 shutdown_connections 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 1 -- 172.21.15.22:0/198107472 wait complete. 2024-03-25T18:28:19.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:19.273+0000 7f49c51f4640 1 librados: shutdown 2024-03-25T18:28:19.286 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:26.093 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:26.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.208+0000 7fe4dec82640 10 monclient: get_monmap_and_config 2024-03-25T18:28:26.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.208+0000 7fe4dec82640 10 monclient: build_initial_monmap 2024-03-25T18:28:26.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.208+0000 7fe4dec82640 10 monclient: monmap: 2024-03-25T18:28:26.209 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:26.209478+0000 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:26.209478+0000 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:26.210 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:26.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 1 Processor -- start 2024-03-25T18:28:26.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 1 -- start start 2024-03-25T18:28:26.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 10 monclient: init 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d80b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d80b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d814be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d80af940 con 0x7fe4d814be90 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d80b0a70 con 0x7fe4d80b4d60 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d80b0d50 con 0x7fe4d80b1730 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.209+0000 7fe4dec82640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:26.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.210+0000 7fe4dec82640 10 monclient(hunting): authenticate will time out at 7705.433583s 2024-03-25T18:28:26.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.211+0000 7fe4dd21f640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fe4d814be90 0x7fe4d814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44774/0 (socket says 172.21.15.22:44774) 2024-03-25T18:28:26.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.211+0000 7fe4dd21f640 1 -- 172.21.15.22:0/2683137989 learned_addr learned my addr 172.21.15.22:0/2683137989 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:26.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.224+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954890672 0 0) 0x7fe4d80af940 con 0x7fe4d814be90 2024-03-25T18:28:26.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.224+0000 7fe4d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.224+0000 7fe4d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.224+0000 7fe4d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.224+0000 7fe4d77fe640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:28:26.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.224+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4c8003520 con 0x7fe4d814be90 2024-03-25T18:28:26.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.235+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865490423 0 0) 0x7fe4d80b0a70 con 0x7fe4d80b4d60 2024-03-25T18:28:26.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.235+0000 7fe4d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.235+0000 7fe4d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.235+0000 7fe4d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.235+0000 7fe4d77fe640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:28:26.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.235+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d80af940 con 0x7fe4d80b4d60 2024-03-25T18:28:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.239+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474863830 0 0) 0x7fe4d80b0d50 con 0x7fe4d80b1730 2024-03-25T18:28:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.239+0000 7fe4d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.239+0000 7fe4d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.239+0000 7fe4d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.239+0000 7fe4d77fe640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:28:26.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.239+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d80b0a70 con 0x7fe4d80b1730 2024-03-25T18:28:26.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187795789 0 0) 0x7fe4c8003520 con 0x7fe4d814be90 2024-03-25T18:28:26.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 >> v1:172.21.15.22:6790/0 conn(0x7fe4d80b1730 legacy=0x7fe4d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 >> v1:172.21.15.33:6789/0 conn(0x7fe4d80b4d60 legacy=0x7fe4d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 1 monclient: found mon.noname-a 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d80b1030 con 0x7fe4d814be90 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 10 monclient: _finish_auth 0 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe4d0001a10 con 0x7fe4d814be90 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4dec82640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4dec82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4d77fe640 10 monclient: dump: 2024-03-25T18:28:26.244 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:26.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.240+0000 7fe4dec82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:26.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4d0001b70 con 0x7fe4d814be90 2024-03-25T18:28:26.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:26.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4d77fe640 1 -- 172.21.15.22:0/2683137989 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe4d0001f80 con 0x7fe4d814be90 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4d77fe640 10 monclient: dump: 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:26.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4dec82640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:26.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4dec82640 4 set_mon_vals no callback set 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4dec82640 10 monclient: shutdown 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4dec82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.252+0000 7fe4dec82640 1 -- 172.21.15.22:0/2683137989 >> v1:172.21.15.22:6789/0 conn(0x7fe4d814be90 legacy=0x7fe4d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.253+0000 7fe4dec82640 1 -- 172.21.15.22:0/2683137989 shutdown_connections 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.253+0000 7fe4dec82640 1 -- 172.21.15.22:0/2683137989 >> 172.21.15.22:0/2683137989 conn(0x7fe4d80617a0 msgr2=0x7fe4d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.253+0000 7fe4dec82640 1 -- 172.21.15.22:0/2683137989 shutdown_connections 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.253+0000 7fe4dec82640 1 -- 172.21.15.22:0/2683137989 wait complete. 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 10 monclient: build_initial_monmap 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 10 monclient: monmap: 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:26.256 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 1 librados: starting msgr at 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 1 librados: starting objecter 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 1 Processor -- start 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 1 -- start start 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 1 librados: setting wanted keys 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 1 librados: calling monclient init 2024-03-25T18:28:26.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.254+0000 7fe4dec82640 10 monclient: init 2024-03-25T18:28:26.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:26.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 10 monclient(hunting): picked mon.noname-a con 0x7fe4d80b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 10 monclient(hunting): picked mon.noname-b con 0x7fe4d80b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 10 monclient(hunting): picked mon.noname-c con 0x7fe4d814be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d8134a00 con 0x7fe4d80b1730 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d8139050 con 0x7fe4d80b4d60 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d813a330 con 0x7fe4d814be90 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:26.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.255+0000 7fe4dec82640 10 monclient(hunting): authenticate will time out at 7705.479471s 2024-03-25T18:28:26.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.264+0000 7fe4dca1e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fe4d80b1730 0x7fe4d813ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44818/0 (socket says 172.21.15.22:44818) 2024-03-25T18:28:26.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.264+0000 7fe4dca1e640 1 -- 172.21.15.22:0/2969629016 learned_addr learned my addr 172.21.15.22:0/2969629016 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:26.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.266+0000 7fe4d57fa640 10 client.?.objecter ms_handle_connect 0x7fe4d80b4d60 2024-03-25T18:28:26.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.267+0000 7fe4d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:26.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.270+0000 7fe4d57fa640 10 client.?.objecter ms_handle_connect 0x7fe4d80b1730 2024-03-25T18:28:26.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.270+0000 7fe4d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:26.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.277+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755079697 0 0) 0x7fe4d8139050 con 0x7fe4d80b4d60 2024-03-25T18:28:26.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.277+0000 7fe4d57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.277+0000 7fe4d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.277+0000 7fe4d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.277+0000 7fe4d57fa640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:28:26.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.277+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4bc004eb0 con 0x7fe4d80b4d60 2024-03-25T18:28:26.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.280+0000 7fe4d57fa640 10 client.?.objecter ms_handle_connect 0x7fe4d814be90 2024-03-25T18:28:26.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.280+0000 7fe4d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:26.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.283+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605918218 0 0) 0x7fe4d8134a00 con 0x7fe4d80b1730 2024-03-25T18:28:26.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.283+0000 7fe4d57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.283+0000 7fe4d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.283+0000 7fe4d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.283+0000 7fe4d57fa640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:28:26.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.283+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8139050 con 0x7fe4d80b1730 2024-03-25T18:28:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.288+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714847542 0 0) 0x7fe4d813a330 con 0x7fe4d814be90 2024-03-25T18:28:26.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.288+0000 7fe4d57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:28:26.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.288+0000 7fe4d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:26.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.288+0000 7fe4d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:26.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.288+0000 7fe4d57fa640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:28:26.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.288+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4d8134a00 con 0x7fe4d814be90 2024-03-25T18:28:26.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.305+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 565756399 0 0) 0x7fe4d8139050 con 0x7fe4d80b1730 2024-03-25T18:28:26.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.305+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d813a330 con 0x7fe4d80b1730 2024-03-25T18:28:26.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.305+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe4cc0019b0 con 0x7fe4d80b1730 2024-03-25T18:28:26.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.305+0000 7fe4d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.305+0000 7fe4d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:26.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.305+0000 7fe4d57fa640 10 monclient(hunting): dump: 2024-03-25T18:28:26.307 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:26.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.305+0000 7fe4d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.313+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3297374506 0 0) 0x7fe4d8134a00 con 0x7fe4d814be90 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.313+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4d8139050 con 0x7fe4d814be90 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.313+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe4d0001e80 con 0x7fe4d814be90 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.313+0000 7fe4d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.313+0000 7fe4d57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.313+0000 7fe4d57fa640 10 monclient(hunting): dump: 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.315 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.316 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:26.316 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:26.316 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:26.316 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:26.316 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 886441852 0 0) 0x7fe4d813a330 con 0x7fe4d80b1730 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 >> v1:172.21.15.22:6790/0 conn(0x7fe4d814be90 legacy=0x7fe4d81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 >> v1:172.21.15.33:6789/0 conn(0x7fe4d80b4d60 legacy=0x7fe4d8133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 1 monclient: found mon.a 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d813a610 con 0x7fe4d80b1730 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 10 monclient: _finish_auth 0 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:26.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 10 monclient: _renew_subs 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d813a8f0 con 0x7fe4d80b1730 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 10 client.4632.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 10 monclient: _renew_subs 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d8143140 con 0x7fe4d80b1730 2024-03-25T18:28:26.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.316+0000 7fe4dec82640 1 librados: init done 2024-03-25T18:28:26.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.317+0000 7fe4dec82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:26.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.317+0000 7fe4dec82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:26.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.317+0000 7fe4dec82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:28:26.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.317+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d80627b0 con 0x7fe4d80b1730 2024-03-25T18:28:26.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.640+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4cc003610 con 0x7fe4d80b1730 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.640+0000 7fe4d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.640+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe4cc0039a0 con 0x7fe4d80b1730 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.640+0000 7fe4d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.640+0000 7fe4d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.640+0000 7fe4d57fa640 10 monclient: dump: 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:26.642 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:26.643 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:26.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.640+0000 7fe4d6ffd640 4 set_mon_vals no callback set 2024-03-25T18:28:26.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.641+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fe4cc003bc0 con 0x7fe4d80b1730 2024-03-25T18:28:26.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.641+0000 7fe4d7fff640 10 monclient: get_auth_request con 0x7fe4bc06df80 auth_method 0 2024-03-25T18:28:26.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.642+0000 7fe4d57fa640 10 client.4632.objecter ms_handle_connect 0x7fe4bc06df80 2024-03-25T18:28:26.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fe4cc054790 con 0x7fe4d80b1730 2024-03-25T18:28:26.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 10 client.4632.objecter ms_dispatch 0x7fe4d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:26.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 3 client.4632.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:26.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 3 client.4632.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:26.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 20 client.4632.objecter dump_active .. 0 homeless 2024-03-25T18:28:26.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4cc0502a0 con 0x7fe4d80b1730 2024-03-25T18:28:26.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:26.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.650+0000 7fe4d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:26.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.883+0000 7fe4dec82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:26.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.883+0000 7fe4dec82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:26.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.883+0000 7fe4dec82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:28:26.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.883+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4d81391e0 con 0x7fe4d80b1730 2024-03-25T18:28:26.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.965+0000 7fe4d57fa640 1 -- 172.21.15.22:0/2969629016 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fe4d81391e0 con 0x7fe4d80b1730 2024-03-25T18:28:26.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.965+0000 7fe4d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:26.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.965+0000 7fe4d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:26.967 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:26.967 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:26.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 10 librados: watch_flush enter 2024-03-25T18:28:26.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 10 librados: watch_flush exit 2024-03-25T18:28:26.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 20 client.4632.objecter shutdown clearing up homeless session... 2024-03-25T18:28:26.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 10 client.4632.objecter successfully canceled tick 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 >> v1:172.21.15.33:6800/215350958 conn(0x7fe4bc06df80 legacy=0x7fe4bc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 10 monclient: shutdown 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.967+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 >> v1:172.21.15.22:6789/0 conn(0x7fe4d80b1730 legacy=0x7fe4d813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.968+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 shutdown_connections 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.968+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 >> 172.21.15.22:0/2969629016 conn(0x7fe4d80617a0 msgr2=0x7fe4d80b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.968+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 shutdown_connections 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.968+0000 7fe4dec82640 1 -- 172.21.15.22:0/2969629016 wait complete. 2024-03-25T18:28:26.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:26.968+0000 7fe4dec82640 1 librados: shutdown 2024-03-25T18:28:26.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:33.787 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:33.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.901+0000 7f89332f6640 10 monclient: get_monmap_and_config 2024-03-25T18:28:33.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.901+0000 7f89332f6640 10 monclient: build_initial_monmap 2024-03-25T18:28:33.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.901+0000 7f89332f6640 10 monclient: monmap: 2024-03-25T18:28:33.903 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:33.903 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:33.903 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:33.903047+0000 2024-03-25T18:28:33.903 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:33.903047+0000 2024-03-25T18:28:33.904 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:33.904 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:33.904 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:33.904 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:33.904 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:33.904 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:33.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.902+0000 7f89332f6640 1 Processor -- start 2024-03-25T18:28:33.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.902+0000 7f89332f6640 1 -- start start 2024-03-25T18:28:33.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.902+0000 7f89332f6640 10 monclient: init 2024-03-25T18:28:33.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 10 monclient(hunting): picked mon.noname-a con 0x7f892c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 10 monclient(hunting): picked mon.noname-c con 0x7f892c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 10 monclient(hunting): picked mon.noname-b con 0x7f892c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c142520 con 0x7f892c0a98f0 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0b28a0 con 0x7f892c13f710 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0b2a30 con 0x7f892c1430c0 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:33.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.903+0000 7f89332f6640 10 monclient(hunting): authenticate will time out at 7713.126944s 2024-03-25T18:28:33.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.912+0000 7f8930891640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f892c13f710 0x7f892c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:55858/0 (socket says 172.21.15.22:55858) 2024-03-25T18:28:33.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.912+0000 7f8930891640 1 -- 172.21.15.22:0/3072613143 learned_addr learned my addr 172.21.15.22:0/3072613143 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:33.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.927+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121403402 0 0) 0x7f892c0b28a0 con 0x7f892c13f710 2024-03-25T18:28:33.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.927+0000 7f8923fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.927+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.927+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.927+0000 7f8923fff640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:28:33.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.927+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8918003520 con 0x7f892c13f710 2024-03-25T18:28:33.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.935+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289959937 0 0) 0x7f892c0b2a30 con 0x7f892c1430c0 2024-03-25T18:28:33.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.935+0000 7f8923fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.935+0000 7f8923fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.935+0000 7f8923fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.935+0000 7f8923fff640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:28:33.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.936+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c0b28a0 con 0x7f892c1430c0 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376247843 0 0) 0x7f8918003520 con 0x7f892c13f710 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 >> v1:172.21.15.22:6790/0 conn(0x7f892c1430c0 legacy=0x7f892c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 >> v1:172.21.15.22:6789/0 conn(0x7f892c0a98f0 legacy=0x7f892c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 1 monclient: found mon.noname-c 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0b44c0 con 0x7f892c13f710 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f89332f6640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f89332f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:33.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f891c001690 con 0x7f892c13f710 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f8923fff640 10 monclient: dump: 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:33.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.941+0000 7f89332f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:33.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.949+0000 7f8923fff640 1 -- 172.21.15.22:0/3072613143 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f891c0017f0 con 0x7f892c13f710 2024-03-25T18:28:33.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.949+0000 7f8923fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:33.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.949+0000 7f89332f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:33.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.949+0000 7f89332f6640 4 set_mon_vals no callback set 2024-03-25T18:28:33.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.950+0000 7f89332f6640 10 monclient: shutdown 2024-03-25T18:28:33.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.950+0000 7f89332f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:33.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.950+0000 7f89332f6640 1 -- 172.21.15.22:0/3072613143 >> v1:172.21.15.33:6789/0 conn(0x7f892c13f710 legacy=0x7f892c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.950+0000 7f89332f6640 1 -- 172.21.15.22:0/3072613143 shutdown_connections 2024-03-25T18:28:33.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.950+0000 7f89332f6640 1 -- 172.21.15.22:0/3072613143 >> 172.21.15.22:0/3072613143 conn(0x7f892c0b7ea0 msgr2=0x7f892c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:33.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.950+0000 7f89332f6640 1 -- 172.21.15.22:0/3072613143 shutdown_connections 2024-03-25T18:28:33.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.950+0000 7f89332f6640 1 -- 172.21.15.22:0/3072613143 wait complete. 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.951+0000 7f89332f6640 10 monclient: build_initial_monmap 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.951+0000 7f89332f6640 10 monclient: monmap: 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:33.955 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.951+0000 7f89332f6640 1 librados: starting msgr at 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.951+0000 7f89332f6640 1 librados: starting objecter 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.951+0000 7f89332f6640 1 Processor -- start 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 1 -- start start 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 1 librados: setting wanted keys 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 1 librados: calling monclient init 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient: init 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient(hunting): picked mon.noname-b con 0x7f892c0a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient(hunting): picked mon.noname-c con 0x7f892c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient(hunting): picked mon.noname-a con 0x7f892c1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:33.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0aba50 con 0x7f892c1430c0 2024-03-25T18:28:33.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0abd30 con 0x7f892c0a98f0 2024-03-25T18:28:33.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0ac010 con 0x7f892c13f710 2024-03-25T18:28:33.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:33.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.952+0000 7f89332f6640 10 monclient(hunting): authenticate will time out at 7713.176373s 2024-03-25T18:28:33.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.962+0000 7f8930891640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f892c13f710 0x7f892c0ac430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52728/0 (socket says 172.21.15.22:52728) 2024-03-25T18:28:33.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.962+0000 7f8930891640 1 -- 172.21.15.22:0/1257459401 learned_addr learned my addr 172.21.15.22:0/1257459401 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:33.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.964+0000 7f8921ffb640 10 client.?.objecter ms_handle_connect 0x7f892c13f710 2024-03-25T18:28:33.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.964+0000 7f8921ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:33.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.966+0000 7f8921ffb640 10 client.?.objecter ms_handle_connect 0x7f892c0a98f0 2024-03-25T18:28:33.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.966+0000 7f8921ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119050978 0 0) 0x7f892c0abd30 con 0x7f892c0a98f0 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910004eb0 con 0x7f892c0a98f0 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838738193 0 0) 0x7f892c0ac010 con 0x7f892c13f710 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c0abd30 con 0x7f892c13f710 2024-03-25T18:28:33.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 client.?.objecter ms_handle_connect 0x7f892c1430c0 2024-03-25T18:28:33.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.972+0000 7f8921ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:33.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.979+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090071996 0 0) 0x7f892c0aba50 con 0x7f892c1430c0 2024-03-25T18:28:33.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.979+0000 7f8921ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:28:33.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.979+0000 7f8921ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:33.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.979+0000 7f8921ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:33.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.979+0000 7f8921ffb640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:28:33.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.979+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c0ac010 con 0x7f892c1430c0 2024-03-25T18:28:33.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.984+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4206734689 0 0) 0x7f892c0abd30 con 0x7f892c13f710 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.984+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c0aba50 con 0x7f892c13f710 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.984+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f891c001d60 con 0x7f892c13f710 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.984+0000 7f8921ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.984+0000 7f8921ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.984+0000 7f8921ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:33.986 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:33.987 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:33.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.984+0000 7f8921ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:33.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.985+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4248697177 0 0) 0x7f8910004eb0 con 0x7f892c0a98f0 2024-03-25T18:28:33.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.985+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c0abd30 con 0x7f892c0a98f0 2024-03-25T18:28:33.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.985+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f89280018c0 con 0x7f892c0a98f0 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.985+0000 7f8921ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.985+0000 7f8921ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.985+0000 7f8921ffb640 10 monclient(hunting): dump: 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:33.988 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:33.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1178579506 0 0) 0x7f892c0aba50 con 0x7f892c13f710 2024-03-25T18:28:33.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 >> v1:172.21.15.33:6789/0 conn(0x7f892c0a98f0 legacy=0x7f892c1363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 >> v1:172.21.15.22:6789/0 conn(0x7f892c1430c0 legacy=0x7f892c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:33.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:28:33.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 1 monclient: found mon.c 2024-03-25T18:28:33.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:33.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c138fe0 con 0x7f892c13f710 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 10 monclient: _finish_auth 0 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f8921ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 10 monclient: _renew_subs 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c137cf0 con 0x7f892c13f710 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 10 monclient: _renew_subs 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.992+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c138ac0 con 0x7f892c13f710 2024-03-25T18:28:33.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.993+0000 7f89332f6640 1 librados: init done 2024-03-25T18:28:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.993+0000 7f89332f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.993+0000 7f89332f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.993+0000 7f89332f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:33.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:33.993+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c0627b0 con 0x7f892c13f710 2024-03-25T18:28:34.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.326+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f891c004a90 con 0x7f892c13f710 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.326+0000 7f8921ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.326+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f891c001250 con 0x7f892c13f710 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.326+0000 7f8921ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.326+0000 7f8921ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.326+0000 7f8921ffb640 10 monclient: dump: 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:34.328 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:34.329 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:34.329 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:34.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.326+0000 7f89237fe640 4 set_mon_vals no callback set 2024-03-25T18:28:34.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.327+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f891c004f60 con 0x7f892c13f710 2024-03-25T18:28:34.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.327+0000 7f8931092640 10 monclient: get_auth_request con 0x7f891006df70 auth_method 0 2024-03-25T18:28:34.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.328+0000 7f8921ffb640 10 client.4652.objecter ms_handle_connect 0x7f891006df70 2024-03-25T18:28:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.332+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f891c0549c0 con 0x7f892c13f710 2024-03-25T18:28:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.332+0000 7f8921ffb640 10 client.4652.objecter ms_dispatch 0x7f892c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.332+0000 7f8921ffb640 3 client.4652.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.332+0000 7f8921ffb640 3 client.4652.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.333+0000 7f8921ffb640 20 client.4652.objecter dump_active .. 0 homeless 2024-03-25T18:28:34.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.333+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f891c0504d0 con 0x7f892c13f710 2024-03-25T18:28:34.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.333+0000 7f8921ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:34.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.333+0000 7f8921ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.608+0000 7f89332f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.608+0000 7f89332f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.608+0000 7f89332f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.608+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f892c139170 con 0x7f892c13f710 2024-03-25T18:28:34.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.633+0000 7f8921ffb640 1 -- 172.21.15.22:0/1257459401 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f891c052400 con 0x7f892c13f710 2024-03-25T18:28:34.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.633+0000 7f8921ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:34.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.633+0000 7f8921ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:34.636 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:34.636 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 10 librados: watch_flush enter 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 10 librados: watch_flush exit 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 20 client.4652.objecter shutdown clearing up homeless session... 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 10 client.4652.objecter successfully canceled tick 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 >> v1:172.21.15.33:6800/215350958 conn(0x7f891006df70 legacy=0x7f8910070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 10 monclient: shutdown 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 >> v1:172.21.15.22:6790/0 conn(0x7f892c13f710 legacy=0x7f892c0ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 shutdown_connections 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 >> 172.21.15.22:0/1257459401 conn(0x7f892c0b7ea0 msgr2=0x7f892c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:34.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.636+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 shutdown_connections 2024-03-25T18:28:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.637+0000 7f89332f6640 1 -- 172.21.15.22:0/1257459401 wait complete. 2024-03-25T18:28:34.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:34.637+0000 7f89332f6640 1 librados: shutdown 2024-03-25T18:28:34.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:38.982 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:41.644 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.035 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:41.697 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.071 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:09:41.733 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.084 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:41.746 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.100 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:41.718 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.104 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:41.723 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.112 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:41.774 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.147 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:09:41.765 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:39.151 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:09:41.813 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:41.461 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:41.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.577+0000 7ff59a258640 10 monclient: get_monmap_and_config 2024-03-25T18:28:41.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.577+0000 7ff59a258640 10 monclient: build_initial_monmap 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.577+0000 7ff59a258640 10 monclient: monmap: 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:41.578490+0000 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:41.578490+0000 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:41.579 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:41.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 1 Processor -- start 2024-03-25T18:28:41.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 1 -- start start 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient: init 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient(hunting): picked mon.noname-b con 0x7ff5940b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient(hunting): picked mon.noname-a con 0x7ff5940b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient(hunting): picked mon.noname-c con 0x7ff59414be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940af940 con 0x7ff5940b1730 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940b0a70 con 0x7ff59414be90 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940b0d50 con 0x7ff5940b4d60 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:41.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.578+0000 7ff59a258640 10 monclient(hunting): authenticate will time out at 7720.802486s 2024-03-25T18:28:41.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.587+0000 7ff592ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff5940b1730 0x7ff5940b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46266/0 (socket says 172.21.15.22:46266) 2024-03-25T18:28:41.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.587+0000 7ff592ffd640 1 -- 172.21.15.22:0/2691012245 learned_addr learned my addr 172.21.15.22:0/2691012245 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:41.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.603+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258345242 0 0) 0x7ff5940af940 con 0x7ff5940b1730 2024-03-25T18:28:41.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.603+0000 7ff5927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.603+0000 7ff5927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.603+0000 7ff5927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.603+0000 7ff5927fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:28:41.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.603+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff57c003520 con 0x7ff5940b1730 2024-03-25T18:28:41.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.612+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798506454 0 0) 0x7ff5940b0d50 con 0x7ff5940b4d60 2024-03-25T18:28:41.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.612+0000 7ff5927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.612+0000 7ff5927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.612+0000 7ff5927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.612+0000 7ff5927fc640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:28:41.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.612+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5940af940 con 0x7ff5940b4d60 2024-03-25T18:28:41.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.614+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316082361 0 0) 0x7ff5940b0a70 con 0x7ff59414be90 2024-03-25T18:28:41.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.614+0000 7ff5927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.614+0000 7ff5927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.614+0000 7ff5927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.614+0000 7ff5927fc640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:28:41.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.614+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5940b0d50 con 0x7ff59414be90 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2945870138 0 0) 0x7ff5940af940 con 0x7ff5940b4d60 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 >> v1:172.21.15.33:6789/0 conn(0x7ff59414be90 legacy=0x7ff59414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 >> v1:172.21.15.22:6789/0 conn(0x7ff5940b1730 legacy=0x7ff5940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 1 monclient: found mon.noname-b 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:28:41.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5940b1030 con 0x7ff5940b4d60 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff584001670 con 0x7ff5940b4d60 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff5927fc640 10 monclient: dump: 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:41.621 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:41.622 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:41.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff59a258640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:28:41.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.618+0000 7ff59a258640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:41.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5840017d0 con 0x7ff5940b4d60 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff5927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff5927fc640 1 -- 172.21.15.22:0/2691012245 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff584001be0 con 0x7ff5940b4d60 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff5927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff5927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff5927fc640 10 monclient: dump: 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:41.634 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:41.635 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:41.635 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:41.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff59a258640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:41.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.632+0000 7ff59a258640 4 set_mon_vals no callback set 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.633+0000 7ff59a258640 10 monclient: shutdown 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.633+0000 7ff59a258640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.633+0000 7ff59a258640 1 -- 172.21.15.22:0/2691012245 >> v1:172.21.15.22:6790/0 conn(0x7ff5940b4d60 legacy=0x7ff59413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.633+0000 7ff59a258640 1 -- 172.21.15.22:0/2691012245 shutdown_connections 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.633+0000 7ff59a258640 1 -- 172.21.15.22:0/2691012245 >> 172.21.15.22:0/2691012245 conn(0x7ff5940617a0 msgr2=0x7ff594061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.633+0000 7ff59a258640 1 -- 172.21.15.22:0/2691012245 shutdown_connections 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.633+0000 7ff59a258640 1 -- 172.21.15.22:0/2691012245 wait complete. 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 10 monclient: build_initial_monmap 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 10 monclient: monmap: 2024-03-25T18:28:41.637 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 1 librados: starting msgr at 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 1 librados: starting objecter 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 1 Processor -- start 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 1 -- start start 2024-03-25T18:28:41.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 1 librados: setting wanted keys 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 1 librados: calling monclient init 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.634+0000 7ff59a258640 10 monclient: init 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 10 monclient(hunting): picked mon.noname-a con 0x7ff5940b1730 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 10 monclient(hunting): picked mon.noname-c con 0x7ff5940b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 10 monclient(hunting): picked mon.noname-b con 0x7ff59414be90 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940aaa40 con 0x7ff5940b1730 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5941d7950 con 0x7ff59414be90 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5941d7620 con 0x7ff5940b4d60 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:41.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.635+0000 7ff59a258640 10 monclient(hunting): authenticate will time out at 7720.858926s 2024-03-25T18:28:41.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.645+0000 7ff592ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff5940b4d60 0x7ff594137de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57118/0 (socket says 172.21.15.22:57118) 2024-03-25T18:28:41.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.645+0000 7ff592ffd640 1 -- 172.21.15.22:0/1668631171 learned_addr learned my addr 172.21.15.22:0/1668631171 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:41.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.647+0000 7ff57bfff640 10 client.?.objecter ms_handle_connect 0x7ff5940b4d60 2024-03-25T18:28:41.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.647+0000 7ff57bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:41.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.648+0000 7ff57bfff640 10 client.?.objecter ms_handle_connect 0x7ff5940b1730 2024-03-25T18:28:41.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.648+0000 7ff57bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:41.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.658+0000 7ff57bfff640 10 client.?.objecter ms_handle_connect 0x7ff59414be90 2024-03-25T18:28:41.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.658+0000 7ff57bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:41.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.661+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915503564 0 0) 0x7ff5940aaa40 con 0x7ff5940b1730 2024-03-25T18:28:41.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.661+0000 7ff57bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.661+0000 7ff57bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.661+0000 7ff57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.661+0000 7ff57bfff640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:28:41.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.661+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff580004eb0 con 0x7ff5940b1730 2024-03-25T18:28:41.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.665+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052859400 0 0) 0x7ff5941d7950 con 0x7ff59414be90 2024-03-25T18:28:41.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.665+0000 7ff57bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.665+0000 7ff57bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.665+0000 7ff57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.665+0000 7ff57bfff640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:28:41.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.665+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5940aaa40 con 0x7ff59414be90 2024-03-25T18:28:41.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.672+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972478758 0 0) 0x7ff5941d7620 con 0x7ff5940b4d60 2024-03-25T18:28:41.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.672+0000 7ff57bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:41.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.672+0000 7ff57bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:41.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.672+0000 7ff57bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:41.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.672+0000 7ff57bfff640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:28:41.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.672+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5941d7950 con 0x7ff5940b4d60 2024-03-25T18:28:41.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.673+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1035859104 0 0) 0x7ff5940aaa40 con 0x7ff59414be90 2024-03-25T18:28:41.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.673+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5941d7620 con 0x7ff59414be90 2024-03-25T18:28:41.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.673+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff58c001b50 con 0x7ff59414be90 2024-03-25T18:28:41.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.673+0000 7ff57bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.673+0000 7ff57bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.673+0000 7ff57bfff640 10 monclient(hunting): dump: 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:41.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.673+0000 7ff57bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:41.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2155015305 0 0) 0x7ff5941d7620 con 0x7ff59414be90 2024-03-25T18:28:41.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 >> v1:172.21.15.22:6790/0 conn(0x7ff5940b4d60 legacy=0x7ff594137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 >> v1:172.21.15.22:6789/0 conn(0x7ff5940b1730 legacy=0x7ff5941d7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:41.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:41.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 1 monclient: found mon.b 2024-03-25T18:28:41.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:41.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594142760 con 0x7ff59414be90 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 10 monclient: _finish_auth 0 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff57bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 10 monclient: _renew_subs 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff594142b40 con 0x7ff59414be90 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 10 monclient: _renew_subs 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff594142e90 con 0x7ff59414be90 2024-03-25T18:28:41.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.677+0000 7ff59a258640 1 librados: init done 2024-03-25T18:28:41.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.678+0000 7ff59a258640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:41.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.678+0000 7ff59a258640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:41.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.678+0000 7ff59a258640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:41.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.678+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5941d77b0 con 0x7ff59414be90 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.969+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff58c001fe0 con 0x7ff59414be90 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.969+0000 7ff57bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.969+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7ff58c003990 con 0x7ff59414be90 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.969+0000 7ff57bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.969+0000 7ff57bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.969+0000 7ff57bfff640 10 monclient: dump: 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.971 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:41.972 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:41.972 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:41.972 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:41.972 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:41.972 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:41.972 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:41.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.969+0000 7ff591ffb640 4 set_mon_vals no callback set 2024-03-25T18:28:41.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.970+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7ff58c003bb0 con 0x7ff59414be90 2024-03-25T18:28:41.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.971+0000 7ff5937fe640 10 monclient: get_auth_request con 0x7ff58006ddd0 auth_method 0 2024-03-25T18:28:41.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.971+0000 7ff57bfff640 10 client.4654.objecter ms_handle_connect 0x7ff58006ddd0 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7ff58c053660 con 0x7ff59414be90 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 10 client.4654.objecter ms_dispatch 0x7ff594000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 3 client.4654.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 3 client.4654.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff58c04f170 con 0x7ff59414be90 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:41.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:41.976+0000 7ff57bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:42.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.214+0000 7ff59a258640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:42.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.214+0000 7ff59a258640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:42.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.214+0000 7ff59a258640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:42.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.214+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5941d7b00 con 0x7ff59414be90 2024-03-25T18:28:42.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.243+0000 7ff57bfff640 1 -- 172.21.15.22:0/1668631171 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7ff5941d7b00 con 0x7ff59414be90 2024-03-25T18:28:42.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.243+0000 7ff57bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:42.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.243+0000 7ff57bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:42.245 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:42.245 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:42.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 10 librados: watch_flush enter 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 10 librados: watch_flush exit 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 10 client.4654.objecter successfully canceled tick 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 >> v1:172.21.15.33:6800/215350958 conn(0x7ff58006ddd0 legacy=0x7ff580070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 10 monclient: shutdown 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 >> v1:172.21.15.33:6789/0 conn(0x7ff59414be90 legacy=0x7ff5941e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 shutdown_connections 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 >> 172.21.15.22:0/1668631171 conn(0x7ff5940617a0 msgr2=0x7ff5940b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 shutdown_connections 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 1 -- 172.21.15.22:0/1668631171 wait complete. 2024-03-25T18:28:42.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:42.246+0000 7ff59a258640 1 librados: shutdown 2024-03-25T18:28:42.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:49.068 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.183+0000 7fac9563e640 10 monclient: get_monmap_and_config 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.183+0000 7fac9563e640 10 monclient: build_initial_monmap 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.183+0000 7fac9563e640 10 monclient: monmap: 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:49.184500+0000 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:49.184500+0000 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:49.185 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:49.186 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:49.186 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:49.186 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:49.186 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.183+0000 7fac9563e640 1 Processor -- start 2024-03-25T18:28:49.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.183+0000 7fac9563e640 1 -- start start 2024-03-25T18:28:49.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.183+0000 7fac9563e640 10 monclient: init 2024-03-25T18:28:49.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.184+0000 7fac9563e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:49.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.184+0000 7fac9563e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:49.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.184+0000 7fac9563e640 10 monclient(hunting): picked mon.noname-b con 0x7fac900a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:49.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.185+0000 7fac9563e640 10 monclient(hunting): picked mon.noname-a con 0x7fac9013f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:49.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.185+0000 7fac9563e640 10 monclient(hunting): picked mon.noname-c con 0x7fac901430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:49.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.185+0000 7fac9563e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac90142520 con 0x7fac9013f710 2024-03-25T18:28:49.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.185+0000 7fac9563e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac900b28a0 con 0x7fac901430c0 2024-03-25T18:28:49.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.185+0000 7fac9563e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac900b2a30 con 0x7fac900a98f0 2024-03-25T18:28:49.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.185+0000 7fac9563e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:49.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.185+0000 7fac9563e640 10 monclient(hunting): authenticate will time out at 7728.408858s 2024-03-25T18:28:49.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.200+0000 7fac867fc640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fac9013f710 0x7fac901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38160/0 (socket says 172.21.15.22:38160) 2024-03-25T18:28:49.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.200+0000 7fac867fc640 1 -- 172.21.15.22:0/3606486716 learned_addr learned my addr 172.21.15.22:0/3606486716 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:49.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.213+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872304342 0 0) 0x7fac90142520 con 0x7fac9013f710 2024-03-25T18:28:49.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.213+0000 7fac8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.213+0000 7fac8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.213+0000 7fac8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.213+0000 7fac8e7fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:28:49.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.213+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c003520 con 0x7fac9013f710 2024-03-25T18:28:49.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.216+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982932709 0 0) 0x7fac900b28a0 con 0x7fac901430c0 2024-03-25T18:28:49.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.216+0000 7fac8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.216+0000 7fac8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.216+0000 7fac8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.216+0000 7fac8e7fc640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:28:49.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.216+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac90142520 con 0x7fac901430c0 2024-03-25T18:28:49.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.219+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051108662 0 0) 0x7fac900b2a30 con 0x7fac900a98f0 2024-03-25T18:28:49.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.219+0000 7fac8e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.219+0000 7fac8e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.219+0000 7fac8e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.219+0000 7fac8e7fc640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:28:49.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.219+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac900b28a0 con 0x7fac900a98f0 2024-03-25T18:28:49.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1393316560 0 0) 0x7fac7c003520 con 0x7fac9013f710 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 >> v1:172.21.15.22:6790/0 conn(0x7fac900a98f0 legacy=0x7fac900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 >> v1:172.21.15.33:6789/0 conn(0x7fac901430c0 legacy=0x7fac9014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900b44c0 con 0x7fac9013f710 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:49.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fac78001690 con 0x7fac9013f710 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac9563e640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac9563e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac8e7fc640 10 monclient: dump: 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:49.227 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.223+0000 7fac9563e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:49.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac780017f0 con 0x7fac9013f710 2024-03-25T18:28:49.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac8e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac8e7fc640 1 -- 172.21.15.22:0/3606486716 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fac78001c00 con 0x7fac9013f710 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac8e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac8e7fc640 10 monclient: dump: 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:49.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:49.254 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:49.254 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:49.254 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac9563e640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:49.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.249+0000 7fac9563e640 4 set_mon_vals no callback set 2024-03-25T18:28:49.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.250+0000 7fac9563e640 10 monclient: shutdown 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.250+0000 7fac9563e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.250+0000 7fac9563e640 1 -- 172.21.15.22:0/3606486716 >> v1:172.21.15.22:6789/0 conn(0x7fac9013f710 legacy=0x7fac901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.250+0000 7fac9563e640 1 -- 172.21.15.22:0/3606486716 shutdown_connections 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.250+0000 7fac9563e640 1 -- 172.21.15.22:0/3606486716 >> 172.21.15.22:0/3606486716 conn(0x7fac900b7ea0 msgr2=0x7fac900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.250+0000 7fac9563e640 1 -- 172.21.15.22:0/3606486716 shutdown_connections 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.250+0000 7fac9563e640 1 -- 172.21.15.22:0/3606486716 wait complete. 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.251+0000 7fac9563e640 10 monclient: build_initial_monmap 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.251+0000 7fac9563e640 10 monclient: monmap: 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:49.255 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:49.256 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:49.256 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:49.256 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:49.256 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.251+0000 7fac9563e640 1 librados: starting msgr at 2024-03-25T18:28:49.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.251+0000 7fac9563e640 1 librados: starting objecter 2024-03-25T18:28:49.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 1 Processor -- start 2024-03-25T18:28:49.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 1 -- start start 2024-03-25T18:28:49.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 1 librados: setting wanted keys 2024-03-25T18:28:49.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 1 librados: calling monclient init 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient: init 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient(hunting): picked mon.noname-a con 0x7fac900a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient(hunting): picked mon.noname-c con 0x7fac9013f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient(hunting): picked mon.noname-b con 0x7fac901430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac900abac0 con 0x7fac900a98f0 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac900abda0 con 0x7fac901430c0 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac901380e0 con 0x7fac9013f710 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:49.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.252+0000 7fac9563e640 10 monclient(hunting): authenticate will time out at 7728.476369s 2024-03-25T18:28:49.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.258+0000 7fac867fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fac9013f710 0x7fac90138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56582/0 (socket says 172.21.15.22:56582) 2024-03-25T18:28:49.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.258+0000 7fac867fc640 1 -- 172.21.15.22:0/978174405 learned_addr learned my addr 172.21.15.22:0/978174405 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:49.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.258+0000 7fac87fff640 10 client.?.objecter ms_handle_connect 0x7fac9013f710 2024-03-25T18:28:49.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.258+0000 7fac87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:49.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.274+0000 7fac87fff640 10 client.?.objecter ms_handle_connect 0x7fac900a98f0 2024-03-25T18:28:49.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.274+0000 7fac87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:49.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.276+0000 7fac87fff640 10 client.?.objecter ms_handle_connect 0x7fac901430c0 2024-03-25T18:28:49.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.276+0000 7fac87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:49.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.277+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337900176 0 0) 0x7fac901380e0 con 0x7fac9013f710 2024-03-25T18:28:49.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.277+0000 7fac87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.277+0000 7fac87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.277+0000 7fac87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.277+0000 7fac87fff640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:28:49.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.277+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac70004eb0 con 0x7fac9013f710 2024-03-25T18:28:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.287+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805127420 0 0) 0x7fac900abda0 con 0x7fac901430c0 2024-03-25T18:28:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.287+0000 7fac87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.287+0000 7fac87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.287+0000 7fac87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.287+0000 7fac87fff640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:28:49.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.287+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac901380e0 con 0x7fac901430c0 2024-03-25T18:28:49.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.300+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524725490 0 0) 0x7fac900abac0 con 0x7fac900a98f0 2024-03-25T18:28:49.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.300+0000 7fac87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:28:49.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.300+0000 7fac87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:49.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.300+0000 7fac87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:49.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.300+0000 7fac87fff640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:28:49.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.300+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac900abda0 con 0x7fac900a98f0 2024-03-25T18:28:49.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.307+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321031292 0 0) 0x7fac901380e0 con 0x7fac901430c0 2024-03-25T18:28:49.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.307+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac900abac0 con 0x7fac901430c0 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.307+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fac88001b50 con 0x7fac901430c0 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.307+0000 7fac87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.307+0000 7fac87fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.307+0000 7fac87fff640 10 monclient(hunting): dump: 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.309 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.307+0000 7fac87fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.314+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 823841518 0 0) 0x7fac70004eb0 con 0x7fac9013f710 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.314+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac901380e0 con 0x7fac9013f710 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.314+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fac78001bc0 con 0x7fac9013f710 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.314+0000 7fac87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.314+0000 7fac87fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.314+0000 7fac87fff640 10 monclient(hunting): dump: 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:49.316 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:49.317 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.336+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2601216636 0 0) 0x7fac900abda0 con 0x7fac900a98f0 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.336+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac70004eb0 con 0x7fac900a98f0 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.336+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fac80001830 con 0x7fac900a98f0 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.336+0000 7fac87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.336+0000 7fac87fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.336+0000 7fac87fff640 10 monclient(hunting): dump: 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:49.338 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:49.339 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3239626758 0 0) 0x7fac900abac0 con 0x7fac901430c0 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 >> v1:172.21.15.22:6790/0 conn(0x7fac9013f710 legacy=0x7fac90138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 >> v1:172.21.15.22:6789/0 conn(0x7fac900a98f0 legacy=0x7fac900ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 10 monclient: _finish_hunting 0 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 1 monclient: found mon.b 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac90137d90 con 0x7fac901430c0 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 10 monclient: _finish_auth 0 2024-03-25T18:28:49.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 10 monclient: _check_auth_tickets 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 10 monclient: _renew_subs 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac90139240 con 0x7fac901430c0 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 10 monclient: _renew_subs 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac90136830 con 0x7fac901430c0 2024-03-25T18:28:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.339+0000 7fac9563e640 1 librados: init done 2024-03-25T18:28:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.340+0000 7fac9563e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.340+0000 7fac9563e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.340+0000 7fac9563e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.340+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac900627b0 con 0x7fac901430c0 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.638+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac88002060 con 0x7fac901430c0 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.638+0000 7fac87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.638+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fac88003b10 con 0x7fac901430c0 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.638+0000 7fac87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.638+0000 7fac87fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.638+0000 7fac87fff640 10 monclient: dump: 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:49.640 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:49.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.638+0000 7fac8dffb640 4 set_mon_vals no callback set 2024-03-25T18:28:49.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.639+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fac88019b60 con 0x7fac901430c0 2024-03-25T18:28:49.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.639+0000 7fac8effd640 10 monclient: get_auth_request con 0x7fac7006e060 auth_method 0 2024-03-25T18:28:49.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.640+0000 7fac87fff640 10 client.4660.objecter ms_handle_connect 0x7fac7006e060 2024-03-25T18:28:49.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fac880538b0 con 0x7fac901430c0 2024-03-25T18:28:49.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 10 client.4660.objecter ms_dispatch 0x7fac90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:49.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 3 client.4660.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:49.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 3 client.4660.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:49.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 20 client.4660.objecter dump_active .. 0 homeless 2024-03-25T18:28:49.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac8804f1f0 con 0x7fac901430c0 2024-03-25T18:28:49.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:49.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.644+0000 7fac87fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:49.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.890+0000 7fac9563e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.890+0000 7fac9563e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.890+0000 7fac9563e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:28:49.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.890+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac9013f070 con 0x7fac901430c0 2024-03-25T18:28:49.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.917+0000 7fac87fff640 1 -- 172.21.15.22:0/978174405 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fac880512f0 con 0x7fac901430c0 2024-03-25T18:28:49.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.918+0000 7fac87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:49.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.918+0000 7fac87fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:49.920 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:49.920 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 10 librados: watch_flush enter 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 10 librados: watch_flush exit 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 20 client.4660.objecter shutdown clearing up homeless session... 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 10 client.4660.objecter successfully canceled tick 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 >> v1:172.21.15.33:6800/215350958 conn(0x7fac7006e060 legacy=0x7fac70070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 10 monclient: shutdown 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 >> v1:172.21.15.33:6789/0 conn(0x7fac901430c0 legacy=0x7fac901e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 shutdown_connections 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 >> 172.21.15.22:0/978174405 conn(0x7fac900b7ea0 msgr2=0x7fac90074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 shutdown_connections 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 1 -- 172.21.15.22:0/978174405 wait complete. 2024-03-25T18:28:49.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:49.920+0000 7fac9563e640 1 librados: shutdown 2024-03-25T18:28:49.934 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:28:56.742 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.861+0000 7fe9ba323640 10 monclient: get_monmap_and_config 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.861+0000 7fe9ba323640 10 monclient: build_initial_monmap 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.861+0000 7fe9ba323640 10 monclient: monmap: 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:28:56.862763+0000 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:28:56.862763+0000 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:56.863 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:56.864 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:28:56.864 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:28:56.864 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:56.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.862+0000 7fe9ba323640 1 Processor -- start 2024-03-25T18:28:56.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.862+0000 7fe9ba323640 1 -- start start 2024-03-25T18:28:56.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.862+0000 7fe9ba323640 10 monclient: init 2024-03-25T18:28:56.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.862+0000 7fe9ba323640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:56.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.862+0000 7fe9ba323640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:28:56.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.862+0000 7fe9ba323640 10 monclient(hunting): picked mon.noname-a con 0x7fe9b40a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:56.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.862+0000 7fe9ba323640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:56.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.863+0000 7fe9ba323640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b41430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:56.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.863+0000 7fe9ba323640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b4142520 con 0x7fe9b40a98f0 2024-03-25T18:28:56.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.863+0000 7fe9ba323640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b40b28a0 con 0x7fe9b413f710 2024-03-25T18:28:56.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.863+0000 7fe9ba323640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b40b2a30 con 0x7fe9b41430c0 2024-03-25T18:28:56.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.863+0000 7fe9ba323640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:56.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.863+0000 7fe9ba323640 10 monclient(hunting): authenticate will time out at 7736.086642s 2024-03-25T18:28:56.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.869+0000 7fe9b37fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fe9b413f710 0x7fe9b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:41402/0 (socket says 172.21.15.22:41402) 2024-03-25T18:28:56.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.869+0000 7fe9b37fe640 1 -- 172.21.15.22:0/1703157849 learned_addr learned my addr 172.21.15.22:0/1703157849 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:56.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.882+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617571045 0 0) 0x7fe9b40b28a0 con 0x7fe9b413f710 2024-03-25T18:28:56.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.882+0000 7fe9b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.882+0000 7fe9b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.882+0000 7fe9b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.882+0000 7fe9b2ffd640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:28:56.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.882+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe99c003520 con 0x7fe9b413f710 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552920516 0 0) 0x7fe99c003520 con 0x7fe9b413f710 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 >> v1:172.21.15.22:6790/0 conn(0x7fe9b41430c0 legacy=0x7fe9b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 >> v1:172.21.15.22:6789/0 conn(0x7fe9b40a98f0 legacy=0x7fe9b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:28:56.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b40b44c0 con 0x7fe9b413f710 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe9a8001690 con 0x7fe9b413f710 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9ba323640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9b2ffd640 10 monclient: dump: 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:56.898 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:56.899 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:56.899 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:56.899 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:56.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.895+0000 7fe9ba323640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a80017f0 con 0x7fe9b413f710 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9b2ffd640 1 -- 172.21.15.22:0/1703157849 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe9a8001c00 con 0x7fe9b413f710 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9b2ffd640 10 monclient: dump: 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.901 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.902 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:56.902 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:56.902 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:56.902 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:56.902 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:56.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9ba323640 10 monclient: get_monmap_and_config success 2024-03-25T18:28:56.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.899+0000 7fe9ba323640 4 set_mon_vals no callback set 2024-03-25T18:28:56.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.900+0000 7fe9ba323640 10 monclient: shutdown 2024-03-25T18:28:56.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.900+0000 7fe9ba323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.900+0000 7fe9ba323640 1 -- 172.21.15.22:0/1703157849 >> v1:172.21.15.33:6789/0 conn(0x7fe9b413f710 legacy=0x7fe9b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.900+0000 7fe9ba323640 1 -- 172.21.15.22:0/1703157849 shutdown_connections 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.900+0000 7fe9ba323640 1 -- 172.21.15.22:0/1703157849 >> 172.21.15.22:0/1703157849 conn(0x7fe9b40b7ea0 msgr2=0x7fe9b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.900+0000 7fe9ba323640 1 -- 172.21.15.22:0/1703157849 shutdown_connections 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.900+0000 7fe9ba323640 1 -- 172.21.15.22:0/1703157849 wait complete. 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.901+0000 7fe9ba323640 10 monclient: build_initial_monmap 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.901+0000 7fe9ba323640 10 monclient: monmap: 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:28:56.904 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.902+0000 7fe9ba323640 1 librados: starting msgr at 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.902+0000 7fe9ba323640 1 librados: starting objecter 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.902+0000 7fe9ba323640 1 Processor -- start 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.902+0000 7fe9ba323640 1 -- start start 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.902+0000 7fe9ba323640 1 librados: setting wanted keys 2024-03-25T18:28:56.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.902+0000 7fe9ba323640 1 librados: calling monclient init 2024-03-25T18:28:56.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.902+0000 7fe9ba323640 10 monclient: init 2024-03-25T18:28:56.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 10 monclient: _reopen_session rank -1 2024-03-25T18:28:56.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b40a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 10 monclient(hunting): picked mon.noname-a con 0x7fe9b413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b41430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b40abdc0 con 0x7fe9b413f710 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b41d7730 con 0x7fe9b41430c0 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b4138f10 con 0x7fe9b40a98f0 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 10 monclient(hunting): _renew_subs 2024-03-25T18:28:56.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.903+0000 7fe9ba323640 10 monclient(hunting): authenticate will time out at 7736.127170s 2024-03-25T18:28:56.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.912+0000 7fe9b88c0640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fe9b41430c0 0x7fe9b41d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:41418/0 (socket says 172.21.15.22:41418) 2024-03-25T18:28:56.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.912+0000 7fe9b88c0640 1 -- 172.21.15.22:0/4062994047 learned_addr learned my addr 172.21.15.22:0/4062994047 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:28:56.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.914+0000 7fe9b0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9b41430c0 2024-03-25T18:28:56.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.914+0000 7fe9b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.919+0000 7fe9b0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9b40a98f0 2024-03-25T18:28:56.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.919+0000 7fe9b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.923+0000 7fe9b0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9b413f710 2024-03-25T18:28:56.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.923+0000 7fe9b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.928+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729717702 0 0) 0x7fe9b41d7730 con 0x7fe9b41430c0 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.928+0000 7fe9b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.928+0000 7fe9b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.928+0000 7fe9b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.928+0000 7fe9b0ff9640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:28:56.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.928+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a0004eb0 con 0x7fe9b41430c0 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.934+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271978140 0 0) 0x7fe9b4138f10 con 0x7fe9b40a98f0 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.934+0000 7fe9b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.934+0000 7fe9b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.934+0000 7fe9b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.934+0000 7fe9b0ff9640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:28:56.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.934+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b41d7730 con 0x7fe9b40a98f0 2024-03-25T18:28:56.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.945+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048904252 0 0) 0x7fe9b40abdc0 con 0x7fe9b413f710 2024-03-25T18:28:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.945+0000 7fe9b0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:28:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.945+0000 7fe9b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:28:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.945+0000 7fe9b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:28:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.945+0000 7fe9b0ff9640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:28:56.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.945+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b4138f10 con 0x7fe9b413f710 2024-03-25T18:28:56.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.957+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2653959979 0 0) 0x7fe9a0004eb0 con 0x7fe9b41430c0 2024-03-25T18:28:56.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.957+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9b40abdc0 con 0x7fe9b41430c0 2024-03-25T18:28:56.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.957+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe9ac001c90 con 0x7fe9b41430c0 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.957+0000 7fe9b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.957+0000 7fe9b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.957+0000 7fe9b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:56.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.957+0000 7fe9b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.958+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2050030467 0 0) 0x7fe9b41d7730 con 0x7fe9b40a98f0 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.958+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a0004eb0 con 0x7fe9b40a98f0 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.958+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe9a40018b0 con 0x7fe9b40a98f0 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.958+0000 7fe9b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.958+0000 7fe9b0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.958+0000 7fe9b0ff9640 10 monclient(hunting): dump: 2024-03-25T18:28:56.960 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:56.961 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2202691712 0 0) 0x7fe9a0004eb0 con 0x7fe9b40a98f0 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 >> v1:172.21.15.33:6789/0 conn(0x7fe9b41430c0 legacy=0x7fe9b41d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 >> v1:172.21.15.22:6789/0 conn(0x7fe9b413f710 legacy=0x7fe9b41d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 1 monclient: found mon.c 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b41390a0 con 0x7fe9b40a98f0 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9ba323640 5 monclient: authenticate success, global_id 4670 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9ba323640 10 monclient: _renew_subs 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9ba323640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.970+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b41d8320 con 0x7fe9b40a98f0 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.971+0000 7fe9ba323640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.971+0000 7fe9ba323640 10 monclient: _renew_subs 2024-03-25T18:28:56.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.971+0000 7fe9ba323640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:56.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.971+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b41d43c0 con 0x7fe9b40a98f0 2024-03-25T18:28:56.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.971+0000 7fe9ba323640 1 librados: init done 2024-03-25T18:28:56.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.972+0000 7fe9ba323640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.972+0000 7fe9ba323640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:56.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.972+0000 7fe9ba323640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:56.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:56.972+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b40627b0 con 0x7fe9b40a98f0 2024-03-25T18:28:57.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.276+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9a4003610 con 0x7fe9b40a98f0 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.276+0000 7fe9b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.276+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fe9a4003920 con 0x7fe9b40a98f0 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.276+0000 7fe9b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.276+0000 7fe9b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.276+0000 7fe9b0ff9640 10 monclient: dump: 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:28:57.278 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:28:57.279 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:28:57.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.276+0000 7fe9b27fc640 4 set_mon_vals no callback set 2024-03-25T18:28:57.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.277+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fe9a40198f0 con 0x7fe9b40a98f0 2024-03-25T18:28:57.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.277+0000 7fe9b37fe640 10 monclient: get_auth_request con 0x7fe9a006dfb0 auth_method 0 2024-03-25T18:28:57.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.278+0000 7fe9b0ff9640 10 client.4670.objecter ms_handle_connect 0x7fe9a006dfb0 2024-03-25T18:28:57.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.283+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7fe9a4053560 con 0x7fe9b40a98f0 2024-03-25T18:28:57.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.283+0000 7fe9b0ff9640 10 client.4670.objecter ms_dispatch 0x7fe9b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:28:57.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.283+0000 7fe9b0ff9640 3 client.4670.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:28:57.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.283+0000 7fe9b0ff9640 3 client.4670.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:28:57.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.283+0000 7fe9b0ff9640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-25T18:28:57.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.283+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9a404f070 con 0x7fe9b40a98f0 2024-03-25T18:28:57.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.284+0000 7fe9b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:28:57.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.284+0000 7fe9b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:28:57.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.533+0000 7fe9ba323640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.533+0000 7fe9ba323640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.533+0000 7fe9ba323640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:28:57.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.533+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9b4069f40 con 0x7fe9b40a98f0 2024-03-25T18:28:57.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.572+0000 7fe9b0ff9640 1 -- 172.21.15.22:0/4062994047 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7fe9a4050fa0 con 0x7fe9b40a98f0 2024-03-25T18:28:57.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.572+0000 7fe9b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:28:57.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.572+0000 7fe9b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:28:57.574 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:28:57.574 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 10 librados: watch_flush enter 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 10 librados: watch_flush exit 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 10 client.4670.objecter successfully canceled tick 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 >> v1:172.21.15.33:6800/215350958 conn(0x7fe9a006dfb0 legacy=0x7fe9a0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 10 monclient: shutdown 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 >> v1:172.21.15.22:6790/0 conn(0x7fe9b40a98f0 legacy=0x7fe9b4138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 shutdown_connections 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 >> 172.21.15.22:0/4062994047 conn(0x7fe9b40b7ea0 msgr2=0x7fe9b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 shutdown_connections 2024-03-25T18:28:57.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 1 -- 172.21.15.22:0/4062994047 wait complete. 2024-03-25T18:28:57.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:28:57.574+0000 7fe9ba323640 1 librados: shutdown 2024-03-25T18:28:57.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:04.092 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:06.754 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.108 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:06.770 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.141 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:06.803 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.152 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:06.770 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.156 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:06.774 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.171 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:06.833 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.178 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:06.840 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.195 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:06.813 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.213 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:06.875 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:04.395 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:04.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.508+0000 7f73111a3640 10 monclient: get_monmap_and_config 2024-03-25T18:29:04.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.508+0000 7f73111a3640 10 monclient: build_initial_monmap 2024-03-25T18:29:04.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.508+0000 7f73111a3640 10 monclient: monmap: 2024-03-25T18:29:04.510 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:04.510 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:04.510230+0000 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:04.510230+0000 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:04.511 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.509+0000 7f73111a3640 1 Processor -- start 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.509+0000 7f73111a3640 1 -- start start 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.509+0000 7f73111a3640 10 monclient: init 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 10 monclient(hunting): picked mon.noname-c con 0x7f730c0b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 10 monclient(hunting): picked mon.noname-b con 0x7f730c0b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 10 monclient(hunting): picked mon.noname-a con 0x7f730c14be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0af940 con 0x7f730c14be90 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0b0a70 con 0x7f730c0b4d60 2024-03-25T18:29:04.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0b0d50 con 0x7f730c0b1730 2024-03-25T18:29:04.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:04.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.510+0000 7f73111a3640 10 monclient(hunting): authenticate will time out at 7743.734185s 2024-03-25T18:29:04.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.518+0000 7f730b59e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f730c14be90 0x7f730c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:32856/0 (socket says 172.21.15.22:32856) 2024-03-25T18:29:04.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.518+0000 7f730b59e640 1 -- 172.21.15.22:0/713059036 learned_addr learned my addr 172.21.15.22:0/713059036 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:04.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.535+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242297824 0 0) 0x7f730c0b0a70 con 0x7f730c0b4d60 2024-03-25T18:29:04.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.535+0000 7f7309d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.535+0000 7f7309d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.535+0000 7f7309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.535+0000 7f7309d9b640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:29:04.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.535+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f4003520 con 0x7f730c0b4d60 2024-03-25T18:29:04.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.539+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1366163268 0 0) 0x7f730c0af940 con 0x7f730c14be90 2024-03-25T18:29:04.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.539+0000 7f7309d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.539+0000 7f7309d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.539+0000 7f7309d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.539+0000 7f7309d9b640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:29:04.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.539+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f730c0b0a70 con 0x7f730c14be90 2024-03-25T18:29:04.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086393407 0 0) 0x7f730c0b0a70 con 0x7f730c14be90 2024-03-25T18:29:04.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 >> v1:172.21.15.22:6790/0 conn(0x7f730c0b1730 legacy=0x7f730c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 >> v1:172.21.15.33:6789/0 conn(0x7f730c0b4d60 legacy=0x7f730c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:29:04.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 1 monclient: found mon.noname-a 2024-03-25T18:29:04.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:04.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:29:04.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0b1030 con 0x7f730c14be90 2024-03-25T18:29:04.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 10 monclient: _finish_auth 0 2024-03-25T18:29:04.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:29:04.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:04.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f73040019f0 con 0x7f730c14be90 2024-03-25T18:29:04.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:04.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f7309d9b640 10 monclient: dump: 2024-03-25T18:29:04.554 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:04.555 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:04.555 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.555 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.555 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.555 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:04.555 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:04.555 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:04.556 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:04.556 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:04.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f73111a3640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:29:04.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.549+0000 7f73111a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:04.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7304001b50 con 0x7f730c14be90 2024-03-25T18:29:04.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f7309d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:04.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f7309d9b640 1 -- 172.21.15.22:0/713059036 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7304001f20 con 0x7f730c14be90 2024-03-25T18:29:04.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f7309d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f7309d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:04.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f7309d9b640 10 monclient: dump: 2024-03-25T18:29:04.563 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:04.564 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:04.565 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:04.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f73111a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:04.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.561+0000 7f73111a3640 4 set_mon_vals no callback set 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.562+0000 7f73111a3640 10 monclient: shutdown 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.562+0000 7f73111a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.562+0000 7f73111a3640 1 -- 172.21.15.22:0/713059036 >> v1:172.21.15.22:6789/0 conn(0x7f730c14be90 legacy=0x7f730c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.562+0000 7f73111a3640 1 -- 172.21.15.22:0/713059036 shutdown_connections 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.562+0000 7f73111a3640 1 -- 172.21.15.22:0/713059036 >> 172.21.15.22:0/713059036 conn(0x7f730c0617a0 msgr2=0x7f730c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.562+0000 7f73111a3640 1 -- 172.21.15.22:0/713059036 shutdown_connections 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.562+0000 7f73111a3640 1 -- 172.21.15.22:0/713059036 wait complete. 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.563+0000 7f73111a3640 10 monclient: build_initial_monmap 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.563+0000 7f73111a3640 10 monclient: monmap: 2024-03-25T18:29:04.568 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.563+0000 7f73111a3640 1 librados: starting msgr at 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.563+0000 7f73111a3640 1 librados: starting objecter 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 1 Processor -- start 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 1 -- start start 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 1 librados: setting wanted keys 2024-03-25T18:29:04.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 1 librados: calling monclient init 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient: init 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient(hunting): picked mon.noname-b con 0x7f730c0b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient(hunting): picked mon.noname-c con 0x7f730c0b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient(hunting): picked mon.noname-a con 0x7f730c14be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c13bfd0 con 0x7f730c14be90 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c1d7af0 con 0x7f730c0b1730 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c13a580 con 0x7f730c0b4d60 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:04.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.564+0000 7f73111a3640 10 monclient(hunting): authenticate will time out at 7743.788503s 2024-03-25T18:29:04.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.570+0000 7f730b59e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f730c14be90 0x7f730c1370c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:32894/0 (socket says 172.21.15.22:32894) 2024-03-25T18:29:04.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.570+0000 7f730b59e640 1 -- 172.21.15.22:0/182120747 learned_addr learned my addr 172.21.15.22:0/182120747 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:04.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.571+0000 7f72f37fe640 10 client.?.objecter ms_handle_connect 0x7f730c14be90 2024-03-25T18:29:04.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.571+0000 7f72f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.579+0000 7f72f37fe640 10 client.?.objecter ms_handle_connect 0x7f730c0b1730 2024-03-25T18:29:04.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.579+0000 7f72f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.580+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482906455 0 0) 0x7f730c13bfd0 con 0x7f730c14be90 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.580+0000 7f72f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.580+0000 7f72f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.580+0000 7f72f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.580+0000 7f72f37fe640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:29:04.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.580+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f8004eb0 con 0x7f730c14be90 2024-03-25T18:29:04.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.583+0000 7f72f37fe640 10 client.?.objecter ms_handle_connect 0x7f730c0b4d60 2024-03-25T18:29:04.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.583+0000 7f72f37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.589+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824744254 0 0) 0x7f72f8004eb0 con 0x7f730c14be90 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.589+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f730c13bfd0 con 0x7f730c14be90 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.589+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7304001f20 con 0x7f730c14be90 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.589+0000 7f72f37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.589+0000 7f72f37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.589+0000 7f72f37fe640 10 monclient(hunting): dump: 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.591 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:04.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.589+0000 7f72f37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.597+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691468075 0 0) 0x7f730c1d7af0 con 0x7f730c0b1730 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.597+0000 7f72f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.597+0000 7f72f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.597+0000 7f72f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.597+0000 7f72f37fe640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:29:04.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.597+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f8004eb0 con 0x7f730c0b1730 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.598+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497054685 0 0) 0x7f730c13a580 con 0x7f730c0b4d60 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.598+0000 7f72f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.598+0000 7f72f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.598+0000 7f72f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.598+0000 7f72f37fe640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:29:04.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.598+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f730c1d7af0 con 0x7f730c0b4d60 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 440820875 0 0) 0x7f730c13bfd0 con 0x7f730c14be90 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 >> v1:172.21.15.22:6790/0 conn(0x7f730c0b4d60 legacy=0x7f730c1d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 >> v1:172.21.15.33:6789/0 conn(0x7f730c0b1730 legacy=0x7f730c13af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:04.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 1 monclient: found mon.a 2024-03-25T18:29:04.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:04.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:04.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c13a860 con 0x7f730c14be90 2024-03-25T18:29:04.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:04.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:04.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f72f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:04.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:29:04.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 10 monclient: _renew_subs 2024-03-25T18:29:04.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:04.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c1d7730 con 0x7f730c14be90 2024-03-25T18:29:04.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 10 client.4662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:04.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 10 monclient: _renew_subs 2024-03-25T18:29:04.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:04.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c1379b0 con 0x7f730c14be90 2024-03-25T18:29:04.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.599+0000 7f73111a3640 1 librados: init done 2024-03-25T18:29:04.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.600+0000 7f73111a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.600+0000 7f73111a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.600+0000 7f73111a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:04.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.600+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c0627b0 con 0x7f730c14be90 2024-03-25T18:29:04.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.966+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73040015d0 con 0x7f730c14be90 2024-03-25T18:29:04.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.966+0000 7f72f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:04.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.966+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7304004fb0 con 0x7f730c14be90 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.966+0000 7f72f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.966+0000 7f72f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.966+0000 7f72f37fe640 10 monclient: dump: 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.966+0000 7f730959a640 4 set_mon_vals no callback set 2024-03-25T18:29:04.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.967+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f730401afc0 con 0x7f730c14be90 2024-03-25T18:29:04.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.968+0000 7f730ad9d640 10 monclient: get_auth_request con 0x7f72f806de20 auth_method 0 2024-03-25T18:29:04.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.968+0000 7f72f37fe640 10 client.4662.objecter ms_handle_connect 0x7f72f806de20 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7f7304054c00 con 0x7f730c14be90 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 10 client.4662.objecter ms_dispatch 0x7f730c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 3 client.4662.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 3 client.4662.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 20 client.4662.objecter dump_active .. 0 homeless 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7304050540 con 0x7f730c14be90 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:04.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:04.973+0000 7f72f37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:05.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.195+0000 7f73111a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:05.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.195+0000 7f73111a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:05.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.195+0000 7f73111a3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:05.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.195+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f730c1d7ca0 con 0x7f730c14be90 2024-03-25T18:29:05.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.258+0000 7f72f37fe640 1 -- 172.21.15.22:0/182120747 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7f730c1d7ca0 con 0x7f730c14be90 2024-03-25T18:29:05.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.258+0000 7f72f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:05.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.258+0000 7f72f37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:05.260 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:05.260 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 10 librados: watch_flush enter 2024-03-25T18:29:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 10 librados: watch_flush exit 2024-03-25T18:29:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 20 client.4662.objecter shutdown clearing up homeless session... 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 10 client.4662.objecter successfully canceled tick 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 >> v1:172.21.15.33:6800/215350958 conn(0x7f72f806de20 legacy=0x7f72f80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 10 monclient: shutdown 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.260+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 >> v1:172.21.15.22:6789/0 conn(0x7f730c14be90 legacy=0x7f730c1370c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.261+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 shutdown_connections 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.261+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 >> 172.21.15.22:0/182120747 conn(0x7f730c0617a0 msgr2=0x7f730c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.261+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 shutdown_connections 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.261+0000 7f73111a3640 1 -- 172.21.15.22:0/182120747 wait complete. 2024-03-25T18:29:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:05.261+0000 7f73111a3640 1 librados: shutdown 2024-03-25T18:29:05.274 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:12.082 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.196+0000 7faa3b41e640 10 monclient: get_monmap_and_config 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.196+0000 7faa3b41e640 10 monclient: build_initial_monmap 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.196+0000 7faa3b41e640 10 monclient: monmap: 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:12.197177+0000 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:12.197177+0000 2024-03-25T18:29:12.197 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:12.198 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:12.198 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:12.198 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:12.198 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:12.198 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.197+0000 7faa3b41e640 1 Processor -- start 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.197+0000 7faa3b41e640 1 -- start start 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.197+0000 7faa3b41e640 10 monclient: init 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 10 monclient(hunting): picked mon.noname-b con 0x7faa340a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 10 monclient(hunting): picked mon.noname-a con 0x7faa3413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:12.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 10 monclient(hunting): picked mon.noname-c con 0x7faa341430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:12.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa34142520 con 0x7faa3413f710 2024-03-25T18:29:12.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa340b28a0 con 0x7faa341430c0 2024-03-25T18:29:12.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa340b2a30 con 0x7faa340a98f0 2024-03-25T18:29:12.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:12.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.198+0000 7faa3b41e640 10 monclient(hunting): authenticate will time out at 7751.420981s 2024-03-25T18:29:12.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.202+0000 7faa389b9640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7faa3413f710 0x7faa341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:37426/0 (socket says 172.21.15.22:37426) 2024-03-25T18:29:12.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.202+0000 7faa389b9640 1 -- 172.21.15.22:0/1648183744 learned_addr learned my addr 172.21.15.22:0/1648183744 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:12.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.218+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804081537 0 0) 0x7faa34142520 con 0x7faa3413f710 2024-03-25T18:29:12.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.218+0000 7faa2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.218+0000 7faa2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.218+0000 7faa2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.218+0000 7faa2bfff640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:29:12.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.218+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa20003520 con 0x7faa3413f710 2024-03-25T18:29:12.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.227+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321273271 0 0) 0x7faa340b28a0 con 0x7faa341430c0 2024-03-25T18:29:12.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.227+0000 7faa2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.227+0000 7faa2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.227+0000 7faa2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.227+0000 7faa2bfff640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:29:12.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.227+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa34142520 con 0x7faa341430c0 2024-03-25T18:29:12.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.228+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915689096 0 0) 0x7faa340b2a30 con 0x7faa340a98f0 2024-03-25T18:29:12.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.228+0000 7faa2bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.228+0000 7faa2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.228+0000 7faa2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.228+0000 7faa2bfff640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:29:12.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.228+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa340b28a0 con 0x7faa340a98f0 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.244+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2184996783 0 0) 0x7faa20003520 con 0x7faa3413f710 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.244+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 >> v1:172.21.15.22:6790/0 conn(0x7faa340a98f0 legacy=0x7faa340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.244+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 >> v1:172.21.15.33:6789/0 conn(0x7faa341430c0 legacy=0x7faa3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.244+0000 7faa2bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 1 monclient: found mon.noname-a 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa340b44c0 con 0x7faa3413f710 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 10 monclient: _finish_auth 0 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa24001690 con 0x7faa3413f710 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa2bfff640 10 monclient: dump: 2024-03-25T18:29:12.246 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa3b41e640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:29:12.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.245+0000 7faa3b41e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:12.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa240017f0 con 0x7faa3413f710 2024-03-25T18:29:12.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:12.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa2bfff640 1 -- 172.21.15.22:0/1648183744 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa24001c00 con 0x7faa3413f710 2024-03-25T18:29:12.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa3b41e640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa3b41e640 4 set_mon_vals no callback set 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.258+0000 7faa2bfff640 10 monclient: dump: 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:12.259 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:12.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.259+0000 7faa3b41e640 10 monclient: shutdown 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.259+0000 7faa3b41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.259+0000 7faa3b41e640 1 -- 172.21.15.22:0/1648183744 >> v1:172.21.15.22:6789/0 conn(0x7faa3413f710 legacy=0x7faa341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.259+0000 7faa3b41e640 1 -- 172.21.15.22:0/1648183744 shutdown_connections 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.259+0000 7faa3b41e640 1 -- 172.21.15.22:0/1648183744 >> 172.21.15.22:0/1648183744 conn(0x7faa340b7ea0 msgr2=0x7faa340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.259+0000 7faa3b41e640 1 -- 172.21.15.22:0/1648183744 shutdown_connections 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.259+0000 7faa3b41e640 1 -- 172.21.15.22:0/1648183744 wait complete. 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 10 monclient: build_initial_monmap 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 10 monclient: monmap: 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:12.261 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 1 librados: starting msgr at 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 1 librados: starting objecter 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 1 Processor -- start 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 1 -- start start 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 1 librados: setting wanted keys 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 1 librados: calling monclient init 2024-03-25T18:29:12.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.260+0000 7faa3b41e640 10 monclient: init 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 10 monclient(hunting): picked mon.noname-b con 0x7faa340a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 10 monclient(hunting): picked mon.noname-c con 0x7faa3413f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 10 monclient(hunting): picked mon.noname-a con 0x7faa341430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa341d82d0 con 0x7faa341430c0 2024-03-25T18:29:12.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa340abb90 con 0x7faa340a98f0 2024-03-25T18:29:12.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa341d76a0 con 0x7faa3413f710 2024-03-25T18:29:12.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:12.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.261+0000 7faa3b41e640 10 monclient(hunting): authenticate will time out at 7751.483922s 2024-03-25T18:29:12.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.263+0000 7faa391ba640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7faa340a98f0 0x7faa341385d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:48772/0 (socket says 172.21.15.22:48772) 2024-03-25T18:29:12.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.263+0000 7faa391ba640 1 -- 172.21.15.22:0/2932781796 learned_addr learned my addr 172.21.15.22:0/2932781796 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:12.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.266+0000 7faa29ffb640 10 client.?.objecter ms_handle_connect 0x7faa340a98f0 2024-03-25T18:29:12.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.266+0000 7faa29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:12.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.275+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2966741538 0 0) 0x7faa340abb90 con 0x7faa340a98f0 2024-03-25T18:29:12.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.275+0000 7faa29ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.275+0000 7faa29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.275+0000 7faa29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.275+0000 7faa29ffb640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:29:12.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.275+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa18004eb0 con 0x7faa340a98f0 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.280+0000 7faa29ffb640 10 client.?.objecter ms_handle_connect 0x7faa341430c0 2024-03-25T18:29:12.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.280+0000 7faa29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.285+0000 7faa29ffb640 10 client.?.objecter ms_handle_connect 0x7faa3413f710 2024-03-25T18:29:12.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.285+0000 7faa29ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.289+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2003887436 0 0) 0x7faa18004eb0 con 0x7faa340a98f0 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.289+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa340abb90 con 0x7faa340a98f0 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.289+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa30001a10 con 0x7faa340a98f0 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.289+0000 7faa29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.289+0000 7faa29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.289+0000 7faa29ffb640 10 monclient(hunting): dump: 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:12.290 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.289+0000 7faa29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.290+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625423839 0 0) 0x7faa341d82d0 con 0x7faa341430c0 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.290+0000 7faa29ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.290+0000 7faa29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.290+0000 7faa29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.290+0000 7faa29ffb640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:29:12.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.290+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa18004eb0 con 0x7faa341430c0 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.293+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973286101 0 0) 0x7faa341d76a0 con 0x7faa3413f710 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.293+0000 7faa29ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.293+0000 7faa29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.293+0000 7faa29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.294+0000 7faa29ffb640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:29:12.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.294+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa341d82d0 con 0x7faa3413f710 2024-03-25T18:29:12.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 209645279 0 0) 0x7faa340abb90 con 0x7faa340a98f0 2024-03-25T18:29:12.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 >> v1:172.21.15.22:6790/0 conn(0x7faa3413f710 legacy=0x7faa341d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 >> v1:172.21.15.22:6789/0 conn(0x7faa341430c0 legacy=0x7faa34134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 1 monclient: found mon.b 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa34134720 con 0x7faa340a98f0 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 10 monclient: _renew_subs 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa34134970 con 0x7faa340a98f0 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 10 client.4678.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 10 monclient: _renew_subs 2024-03-25T18:29:12.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:12.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa34137a10 con 0x7faa340a98f0 2024-03-25T18:29:12.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.299+0000 7faa3b41e640 1 librados: init done 2024-03-25T18:29:12.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.300+0000 7faa3b41e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:12.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.300+0000 7faa3b41e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:12.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.300+0000 7faa3b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:12.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.300+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa341d7830 con 0x7faa340a98f0 2024-03-25T18:29:12.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.588+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa30001cb0 con 0x7faa340a98f0 2024-03-25T18:29:12.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.588+0000 7faa29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:12.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.588+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7faa30003a60 con 0x7faa340a98f0 2024-03-25T18:29:12.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.588+0000 7faa29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:12.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.588+0000 7faa29ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:12.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.588+0000 7faa29ffb640 10 monclient: dump: 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:12.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.588+0000 7faa2b7fe640 4 set_mon_vals no callback set 2024-03-25T18:29:12.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.589+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7faa30003c80 con 0x7faa340a98f0 2024-03-25T18:29:12.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.590+0000 7faa389b9640 10 monclient: get_auth_request con 0x7faa1806de80 auth_method 0 2024-03-25T18:29:12.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.590+0000 7faa29ffb640 10 client.4678.objecter ms_handle_connect 0x7faa1806de80 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.595+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1175319491 0 0) 0x7faa300548a0 con 0x7faa340a98f0 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.595+0000 7faa29ffb640 10 client.4678.objecter ms_dispatch 0x7faa34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.595+0000 7faa29ffb640 3 client.4678.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.595+0000 7faa29ffb640 3 client.4678.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.595+0000 7faa29ffb640 20 client.4678.objecter dump_active .. 0 homeless 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.596+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa300503b0 con 0x7faa340a98f0 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.596+0000 7faa29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:12.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.596+0000 7faa29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:12.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.850+0000 7faa3b41e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.850+0000 7faa3b41e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.850+0000 7faa3b41e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:12.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.850+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa341d7490 con 0x7faa340a98f0 2024-03-25T18:29:12.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.885+0000 7faa29ffb640 1 -- 172.21.15.22:0/2932781796 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1101949412) 0x7faa300522e0 con 0x7faa340a98f0 2024-03-25T18:29:12.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.885+0000 7faa29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:12.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.885+0000 7faa29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:12.886 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:12.886 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:19:20.216189+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:19:20.216189+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:12.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.887+0000 7faa3b41e640 10 librados: watch_flush enter 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.887+0000 7faa3b41e640 10 librados: watch_flush exit 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.887+0000 7faa3b41e640 20 client.4678.objecter shutdown clearing up homeless session... 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.887+0000 7faa3b41e640 10 client.4678.objecter successfully canceled tick 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 >> v1:172.21.15.33:6800/215350958 conn(0x7faa1806de80 legacy=0x7faa18070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 10 monclient: shutdown 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 >> v1:172.21.15.33:6789/0 conn(0x7faa340a98f0 legacy=0x7faa341385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 shutdown_connections 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 >> 172.21.15.22:0/2932781796 conn(0x7faa340b7ea0 msgr2=0x7faa34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 shutdown_connections 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 1 -- 172.21.15.22:0/2932781796 wait complete. 2024-03-25T18:29:12.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:12.888+0000 7faa3b41e640 1 librados: shutdown 2024-03-25T18:29:12.900 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:19.706 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:19.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.820+0000 7f02fad4d640 10 monclient: get_monmap_and_config 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.820+0000 7f02fad4d640 10 monclient: build_initial_monmap 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.820+0000 7f02fad4d640 10 monclient: monmap: 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:19.820741+0000 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:19.820741+0000 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:19.821 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:19.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 1 Processor -- start 2024-03-25T18:29:19.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 1 -- start start 2024-03-25T18:29:19.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 10 monclient: init 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 10 monclient(hunting): picked mon.noname-c con 0x7f02f40a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 10 monclient(hunting): picked mon.noname-a con 0x7f02f413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 10 monclient(hunting): picked mon.noname-b con 0x7f02f41430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f4142520 con 0x7f02f413f710 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f40b28a0 con 0x7f02f40a98f0 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f40b2a30 con 0x7f02f41430c0 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.821+0000 7f02fad4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:19.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.822+0000 7f02fad4d640 10 monclient(hunting): authenticate will time out at 7759.044584s 2024-03-25T18:29:19.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.835+0000 7f02f92ea640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f02f41430c0 0x7f02f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:33594/0 (socket says 172.21.15.22:33594) 2024-03-25T18:29:19.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.835+0000 7f02f92ea640 1 -- 172.21.15.22:0/2810970288 learned_addr learned my addr 172.21.15.22:0/2810970288 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:19.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.848+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293878232 0 0) 0x7f02f40b2a30 con 0x7f02f41430c0 2024-03-25T18:29:19.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.848+0000 7f02f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.848+0000 7f02f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.848+0000 7f02f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.848+0000 7f02f37fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:29:19.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.848+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e4003520 con 0x7f02f41430c0 2024-03-25T18:29:19.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.850+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804846988 0 0) 0x7f02f40b28a0 con 0x7f02f40a98f0 2024-03-25T18:29:19.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.850+0000 7f02f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.850+0000 7f02f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.850+0000 7f02f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.850+0000 7f02f37fe640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:29:19.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.850+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f40b2a30 con 0x7f02f40a98f0 2024-03-25T18:29:19.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.857+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2760263789 0 0) 0x7f02f4142520 con 0x7f02f413f710 2024-03-25T18:29:19.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.857+0000 7f02f37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.857+0000 7f02f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.857+0000 7f02f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.857+0000 7f02f37fe640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:29:19.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.857+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f40b28a0 con 0x7f02f413f710 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1007012119 0 0) 0x7f02e4003520 con 0x7f02f41430c0 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 >> v1:172.21.15.33:6789/0 conn(0x7f02f40a98f0 legacy=0x7f02f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 >> v1:172.21.15.22:6789/0 conn(0x7f02f413f710 legacy=0x7f02f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 1 monclient: found mon.noname-b 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:29:19.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40b44c0 con 0x7f02f41430c0 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f02ec001a10 con 0x7f02f41430c0 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02f37fe640 10 monclient: dump: 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:19.872 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:19.873 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:19.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02fad4d640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:29:19.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.870+0000 7f02fad4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02ec001b70 con 0x7f02f41430c0 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02f37fe640 1 -- 172.21.15.22:0/2810970288 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f02ec001f80 con 0x7f02f41430c0 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02f37fe640 10 monclient: dump: 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:19.901 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02fad4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:19.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.900+0000 7f02fad4d640 4 set_mon_vals no callback set 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.901+0000 7f02fad4d640 10 monclient: shutdown 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.901+0000 7f02fad4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.901+0000 7f02fad4d640 1 -- 172.21.15.22:0/2810970288 >> v1:172.21.15.22:6790/0 conn(0x7f02f41430c0 legacy=0x7f02f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.901+0000 7f02fad4d640 1 -- 172.21.15.22:0/2810970288 shutdown_connections 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.901+0000 7f02fad4d640 1 -- 172.21.15.22:0/2810970288 >> 172.21.15.22:0/2810970288 conn(0x7f02f40b7ea0 msgr2=0x7f02f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.901+0000 7f02fad4d640 1 -- 172.21.15.22:0/2810970288 shutdown_connections 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.901+0000 7f02fad4d640 1 -- 172.21.15.22:0/2810970288 wait complete. 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.902+0000 7f02fad4d640 10 monclient: build_initial_monmap 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.902+0000 7f02fad4d640 10 monclient: monmap: 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:19.905 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.903+0000 7f02fad4d640 1 librados: starting msgr at 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.903+0000 7f02fad4d640 1 librados: starting objecter 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.903+0000 7f02fad4d640 1 Processor -- start 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.903+0000 7f02fad4d640 1 -- start start 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.903+0000 7f02fad4d640 1 librados: setting wanted keys 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.903+0000 7f02fad4d640 1 librados: calling monclient init 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.903+0000 7f02fad4d640 10 monclient: init 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:19.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 10 monclient(hunting): picked mon.noname-b con 0x7f02f40a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 10 monclient(hunting): picked mon.noname-a con 0x7f02f413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 10 monclient(hunting): picked mon.noname-c con 0x7f02f41430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f4137190 con 0x7f02f413f710 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f41d41e0 con 0x7f02f40a98f0 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f41d4430 con 0x7f02f41430c0 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.904+0000 7f02fad4d640 10 monclient(hunting): authenticate will time out at 7759.127073s 2024-03-25T18:29:19.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.913+0000 7f02f92ea640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f02f41430c0 0x7f02f4130ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:33644/0 (socket says 172.21.15.22:33644) 2024-03-25T18:29:19.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.913+0000 7f02f92ea640 1 -- 172.21.15.22:0/3693489807 learned_addr learned my addr 172.21.15.22:0/3693489807 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.915+0000 7f02f17fa640 10 client.?.objecter ms_handle_connect 0x7f02f41430c0 2024-03-25T18:29:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.915+0000 7f02f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.918+0000 7f02f17fa640 10 client.?.objecter ms_handle_connect 0x7f02f40a98f0 2024-03-25T18:29:19.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.918+0000 7f02f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.920+0000 7f02f17fa640 10 client.?.objecter ms_handle_connect 0x7f02f413f710 2024-03-25T18:29:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.920+0000 7f02f17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:19.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.932+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274536524 0 0) 0x7f02f4137190 con 0x7f02f413f710 2024-03-25T18:29:19.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.932+0000 7f02f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.932+0000 7f02f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.932+0000 7f02f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.932+0000 7f02f17fa640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:29:19.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.932+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d8004eb0 con 0x7f02f413f710 2024-03-25T18:29:19.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.938+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231313003 0 0) 0x7f02f41d41e0 con 0x7f02f40a98f0 2024-03-25T18:29:19.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.938+0000 7f02f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.938+0000 7f02f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.938+0000 7f02f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.938+0000 7f02f17fa640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:29:19.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.938+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f4137190 con 0x7f02f40a98f0 2024-03-25T18:29:19.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.945+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157906669 0 0) 0x7f02f41d4430 con 0x7f02f41430c0 2024-03-25T18:29:19.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.945+0000 7f02f17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:29:19.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.945+0000 7f02f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:19.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.945+0000 7f02f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:19.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.945+0000 7f02f17fa640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:29:19.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.945+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f41d41e0 con 0x7f02f41430c0 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.958+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024060633 0 0) 0x7f02d8004eb0 con 0x7f02f413f710 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.958+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02f41d4430 con 0x7f02f413f710 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.958+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f02e00018f0 con 0x7f02f413f710 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.958+0000 7f02f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.958+0000 7f02f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.958+0000 7f02f17fa640 10 monclient(hunting): dump: 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:19.959 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:19.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.958+0000 7f02f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:19.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.962+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399273239 0 0) 0x7f02f4137190 con 0x7f02f40a98f0 2024-03-25T18:29:19.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.962+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02d8004eb0 con 0x7f02f40a98f0 2024-03-25T18:29:19.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.962+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f02e8001970 con 0x7f02f40a98f0 2024-03-25T18:29:19.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.962+0000 7f02f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.962+0000 7f02f17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.962+0000 7f02f17fa640 10 monclient(hunting): dump: 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:19.963 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:19.966 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:19.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.969+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3005483867 0 0) 0x7f02f41d41e0 con 0x7f02f41430c0 2024-03-25T18:29:19.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.969+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02f4137190 con 0x7f02f41430c0 2024-03-25T18:29:19.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.969+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f02ec001e80 con 0x7f02f41430c0 2024-03-25T18:29:19.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.969+0000 7f02f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:19.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.969+0000 7f02f17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.969+0000 7f02f17fa640 10 monclient(hunting): dump: 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:19.970 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:19.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1957473882 0 0) 0x7f02d8004eb0 con 0x7f02f40a98f0 2024-03-25T18:29:19.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 >> v1:172.21.15.22:6790/0 conn(0x7f02f41430c0 legacy=0x7f02f4130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 >> v1:172.21.15.22:6789/0 conn(0x7f02f413f710 legacy=0x7f02f41352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:19.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 1 monclient: found mon.b 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f413bf90 con 0x7f02f40a98f0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 10 monclient: _finish_auth 0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 10 monclient: _renew_subs 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f41d5060 con 0x7f02f40a98f0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 10 monclient: _renew_subs 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f4139f00 con 0x7f02f40a98f0 2024-03-25T18:29:19.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.974+0000 7f02fad4d640 1 librados: init done 2024-03-25T18:29:19.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.975+0000 7f02fad4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:19.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.975+0000 7f02fad4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:19.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.975+0000 7f02fad4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:19.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:19.976+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f41d45c0 con 0x7f02f40a98f0 2024-03-25T18:29:20.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.275+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02e80035f0 con 0x7f02f40a98f0 2024-03-25T18:29:20.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.275+0000 7f02f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:20.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.275+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f02e8003900 con 0x7f02f40a98f0 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.275+0000 7f02f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.275+0000 7f02f17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.275+0000 7f02f17fa640 10 monclient: dump: 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:20.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.275+0000 7f02f2ffd640 4 set_mon_vals no callback set 2024-03-25T18:29:20.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.276+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f02e80198d0 con 0x7f02f40a98f0 2024-03-25T18:29:20.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.277+0000 7f02f3fff640 10 monclient: get_auth_request con 0x7f02d806e110 auth_method 0 2024-03-25T18:29:20.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.277+0000 7f02f17fa640 10 client.4684.objecter ms_handle_connect 0x7f02d806e110 2024-03-25T18:29:20.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f02e8053550 con 0x7f02f40a98f0 2024-03-25T18:29:20.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 10 client.4684.objecter ms_dispatch 0x7f02f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:20.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 3 client.4684.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:20.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 3 client.4684.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:20.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T18:29:20.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02e804ee90 con 0x7f02f40a98f0 2024-03-25T18:29:20.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:20.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.282+0000 7f02f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:20.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.549+0000 7f02fad4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:20.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.549+0000 7f02fad4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:20.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.549+0000 7f02fad4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:20.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.549+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02f40627b0 con 0x7f02f40a98f0 2024-03-25T18:29:20.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.568+0000 7f02f17fa640 1 -- 172.21.15.22:0/3693489807 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f02e8050f90 con 0x7f02f40a98f0 2024-03-25T18:29:20.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.568+0000 7f02f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:20.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.568+0000 7f02f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:20.569 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:20.569 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:20.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.570+0000 7f02fad4d640 10 librados: watch_flush enter 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.570+0000 7f02fad4d640 10 librados: watch_flush exit 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.570+0000 7f02fad4d640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.570+0000 7f02fad4d640 10 client.4684.objecter successfully canceled tick 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 >> v1:172.21.15.33:6800/215350958 conn(0x7f02d806e110 legacy=0x7f02d80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 10 monclient: shutdown 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 >> v1:172.21.15.33:6789/0 conn(0x7f02f40a98f0 legacy=0x7f02f41d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 shutdown_connections 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 >> 172.21.15.22:0/3693489807 conn(0x7f02f40b7ea0 msgr2=0x7f02f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 shutdown_connections 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 1 -- 172.21.15.22:0/3693489807 wait complete. 2024-03-25T18:29:20.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:20.571+0000 7f02fad4d640 1 librados: shutdown 2024-03-25T18:29:20.585 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:24.197 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:26.859 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.204 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:26.866 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.248 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:26.909 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.251 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:26.869 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.253 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:26.915 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.259 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:26.877 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.261 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:26.923 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.280 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:26.898 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:24.283 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:26.945 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:27.391 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:27.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.506+0000 7fd999825640 10 monclient: get_monmap_and_config 2024-03-25T18:29:27.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.506+0000 7fd999825640 10 monclient: build_initial_monmap 2024-03-25T18:29:27.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.507+0000 7fd999825640 10 monclient: monmap: 2024-03-25T18:29:27.507 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:27.507 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:27.507299+0000 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:27.507299+0000 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:27.508 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.507+0000 7fd999825640 1 Processor -- start 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.507+0000 7fd999825640 1 -- start start 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.507+0000 7fd999825640 10 monclient: init 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 10 monclient(hunting): picked mon.noname-b con 0x7fd9940a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 10 monclient(hunting): picked mon.noname-c con 0x7fd99413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 10 monclient(hunting): picked mon.noname-a con 0x7fd9941430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd994142520 con 0x7fd9941430c0 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940b28a0 con 0x7fd99413f710 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940b2a30 con 0x7fd9940a98f0 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:27.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.508+0000 7fd999825640 10 monclient(hunting): authenticate will time out at 7766.730965s 2024-03-25T18:29:27.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.514+0000 7fd992ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fd9940a98f0 0x7fd9940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:46634/0 (socket says 172.21.15.22:46634) 2024-03-25T18:29:27.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.514+0000 7fd992ffd640 1 -- 172.21.15.22:0/1824279276 learned_addr learned my addr 172.21.15.22:0/1824279276 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:27.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.524+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1241361090 0 0) 0x7fd9940b2a30 con 0x7fd9940a98f0 2024-03-25T18:29:27.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.524+0000 7fd991ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.524+0000 7fd991ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.524+0000 7fd991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.524+0000 7fd991ffb640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:29:27.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.524+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd97c003520 con 0x7fd9940a98f0 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2801576469 0 0) 0x7fd97c003520 con 0x7fd9940a98f0 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 >> v1:172.21.15.33:6789/0 conn(0x7fd99413f710 legacy=0x7fd9941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 >> v1:172.21.15.22:6789/0 conn(0x7fd9941430c0 legacy=0x7fd99414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 1 monclient: found mon.noname-b 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:29:27.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940b44c0 con 0x7fd9940a98f0 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 10 monclient: _finish_auth 0 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd991ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd999825640 5 monclient: authenticate success, global_id 4691 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.531+0000 7fd999825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.532+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd984001690 con 0x7fd9940a98f0 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.532+0000 7fd991ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.532+0000 7fd991ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.532+0000 7fd991ffb640 10 monclient: dump: 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.533 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.534 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:27.534 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:27.534 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:27.534 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:27.534 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.532+0000 7fd999825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:27.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd984001920 con 0x7fd9940a98f0 2024-03-25T18:29:27.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd991ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:27.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd991ffb640 1 -- 172.21.15.22:0/1824279276 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd984001c30 con 0x7fd9940a98f0 2024-03-25T18:29:27.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd991ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd991ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:27.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd991ffb640 10 monclient: dump: 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd999825640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:27.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd999825640 4 set_mon_vals no callback set 2024-03-25T18:29:27.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd999825640 10 monclient: shutdown 2024-03-25T18:29:27.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd999825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:27.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.539+0000 7fd999825640 1 -- 172.21.15.22:0/1824279276 >> v1:172.21.15.22:6790/0 conn(0x7fd9940a98f0 legacy=0x7fd9940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.540+0000 7fd999825640 1 -- 172.21.15.22:0/1824279276 shutdown_connections 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.540+0000 7fd999825640 1 -- 172.21.15.22:0/1824279276 >> 172.21.15.22:0/1824279276 conn(0x7fd9940b7ea0 msgr2=0x7fd9940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.540+0000 7fd999825640 1 -- 172.21.15.22:0/1824279276 shutdown_connections 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.540+0000 7fd999825640 1 -- 172.21.15.22:0/1824279276 wait complete. 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.540+0000 7fd999825640 10 monclient: build_initial_monmap 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.540+0000 7fd999825640 10 monclient: monmap: 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:27.542 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:27.543 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:27.543 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:27.543 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:27.543 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.541+0000 7fd999825640 1 librados: starting msgr at 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.541+0000 7fd999825640 1 librados: starting objecter 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.541+0000 7fd999825640 1 Processor -- start 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.541+0000 7fd999825640 1 -- start start 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.541+0000 7fd999825640 1 librados: setting wanted keys 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.541+0000 7fd999825640 1 librados: calling monclient init 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.541+0000 7fd999825640 10 monclient: init 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 10 monclient(hunting): picked mon.noname-a con 0x7fd9940a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 10 monclient(hunting): picked mon.noname-b con 0x7fd99413f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 10 monclient(hunting): picked mon.noname-c con 0x7fd9941430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd994139990 con 0x7fd9940a98f0 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9941d7880 con 0x7fd99413f710 2024-03-25T18:29:27.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9941d84b0 con 0x7fd9941430c0 2024-03-25T18:29:27.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:27.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.542+0000 7fd999825640 10 monclient(hunting): authenticate will time out at 7766.765274s 2024-03-25T18:29:27.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.549+0000 7fd9937fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fd9941430c0 0x7fd994138540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:46638/0 (socket says 172.21.15.22:46638) 2024-03-25T18:29:27.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.550+0000 7fd9937fe640 1 -- 172.21.15.22:0/3445618837 learned_addr learned my addr 172.21.15.22:0/3445618837 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:27.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.551+0000 7fd97bfff640 10 client.?.objecter ms_handle_connect 0x7fd9941430c0 2024-03-25T18:29:27.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.551+0000 7fd97bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:27.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.552+0000 7fd97bfff640 10 client.?.objecter ms_handle_connect 0x7fd9940a98f0 2024-03-25T18:29:27.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.552+0000 7fd97bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:27.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.555+0000 7fd97bfff640 10 client.?.objecter ms_handle_connect 0x7fd99413f710 2024-03-25T18:29:27.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.555+0000 7fd97bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.569+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321262063 0 0) 0x7fd994139990 con 0x7fd9940a98f0 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.569+0000 7fd97bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.569+0000 7fd97bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.569+0000 7fd97bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.569+0000 7fd97bfff640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:29:27.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.569+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd980004eb0 con 0x7fd9940a98f0 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.571+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217569895 0 0) 0x7fd9941d84b0 con 0x7fd9941430c0 2024-03-25T18:29:27.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.571+0000 7fd97bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.571+0000 7fd97bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.571+0000 7fd97bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.571+0000 7fd97bfff640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:29:27.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.571+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd994139990 con 0x7fd9941430c0 2024-03-25T18:29:27.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.576+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319050814 0 0) 0x7fd9941d7880 con 0x7fd99413f710 2024-03-25T18:29:27.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.576+0000 7fd97bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:29:27.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.576+0000 7fd97bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:27.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.576+0000 7fd97bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:27.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.576+0000 7fd97bfff640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:29:27.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.576+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9941d84b0 con 0x7fd99413f710 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.589+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2914484369 0 0) 0x7fd980004eb0 con 0x7fd9940a98f0 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.589+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9941d7880 con 0x7fd9940a98f0 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.589+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd984001650 con 0x7fd9940a98f0 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.589+0000 7fd97bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.589+0000 7fd97bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.589+0000 7fd97bfff640 10 monclient(hunting): dump: 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:27.590 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.589+0000 7fd97bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:27.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.597+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 6059942 0 0) 0x7fd994139990 con 0x7fd9941430c0 2024-03-25T18:29:27.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.597+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd980004eb0 con 0x7fd9941430c0 2024-03-25T18:29:27.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.597+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd98c001c10 con 0x7fd9941430c0 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.597+0000 7fd97bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.597+0000 7fd97bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.597+0000 7fd97bfff640 10 monclient(hunting): dump: 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:27.598 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:27.599 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:27.599 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:27.599 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.602+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1351650251 0 0) 0x7fd9941d84b0 con 0x7fd99413f710 2024-03-25T18:29:27.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.602+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd994139990 con 0x7fd99413f710 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.602+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd988001860 con 0x7fd99413f710 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.602+0000 7fd97bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.602+0000 7fd97bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.602+0000 7fd97bfff640 10 monclient(hunting): dump: 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:27.603 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3790187878 0 0) 0x7fd980004eb0 con 0x7fd9941430c0 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 >> v1:172.21.15.33:6789/0 conn(0x7fd99413f710 legacy=0x7fd9941d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 >> v1:172.21.15.22:6789/0 conn(0x7fd9940a98f0 legacy=0x7fd9941d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 1 monclient: found mon.c 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9941d5000 con 0x7fd9941430c0 2024-03-25T18:29:27.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 10 monclient: _finish_auth 0 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd97bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 10 monclient: _renew_subs 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9941d52b0 con 0x7fd9941430c0 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 10 monclient: _renew_subs 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9941d43c0 con 0x7fd9941430c0 2024-03-25T18:29:27.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.606+0000 7fd999825640 1 librados: init done 2024-03-25T18:29:27.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.607+0000 7fd999825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:27.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.607+0000 7fd999825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:27.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.607+0000 7fd999825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:27.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.607+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9941d76e0 con 0x7fd9941430c0 2024-03-25T18:29:27.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.910+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd98c003990 con 0x7fd9941430c0 2024-03-25T18:29:27.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.911+0000 7fd97bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:27.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.911+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fd98c002100 con 0x7fd9941430c0 2024-03-25T18:29:27.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.911+0000 7fd97bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:27.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.911+0000 7fd97bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.911+0000 7fd97bfff640 10 monclient: dump: 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:27.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.911+0000 7fd9917fa640 4 set_mon_vals no callback set 2024-03-25T18:29:27.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.911+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fd98c003c80 con 0x7fd9941430c0 2024-03-25T18:29:27.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.912+0000 7fd992ffd640 10 monclient: get_auth_request con 0x7fd98006e160 auth_method 0 2024-03-25T18:29:27.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.912+0000 7fd97bfff640 10 client.4694.objecter ms_handle_connect 0x7fd98006e160 2024-03-25T18:29:27.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7fd98c0548c0 con 0x7fd9941430c0 2024-03-25T18:29:27.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 10 client.4694.objecter ms_dispatch 0x7fd994000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:27.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:27.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:27.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T18:29:27.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd98c0503d0 con 0x7fd9941430c0 2024-03-25T18:29:27.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:27.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:27.918+0000 7fd97bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:28.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.174+0000 7fd999825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:28.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.174+0000 7fd999825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:28.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.174+0000 7fd999825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:28.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.174+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9941d7a10 con 0x7fd9941430c0 2024-03-25T18:29:28.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.223+0000 7fd97bfff640 1 -- 172.21.15.22:0/3445618837 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7fd98c052300 con 0x7fd9941430c0 2024-03-25T18:29:28.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.223+0000 7fd97bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:28.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.223+0000 7fd97bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:28.224 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:28.224 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:28.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 10 librados: watch_flush enter 2024-03-25T18:29:28.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 10 librados: watch_flush exit 2024-03-25T18:29:28.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T18:29:28.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 10 client.4694.objecter successfully canceled tick 2024-03-25T18:29:28.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 >> v1:172.21.15.33:6800/215350958 conn(0x7fd98006e160 legacy=0x7fd980070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 10 monclient: shutdown 2024-03-25T18:29:28.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:28.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 >> v1:172.21.15.22:6790/0 conn(0x7fd9941430c0 legacy=0x7fd994138540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:28.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 shutdown_connections 2024-03-25T18:29:28.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.226+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 >> 172.21.15.22:0/3445618837 conn(0x7fd9940b7ea0 msgr2=0x7fd994074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:28.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.227+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 shutdown_connections 2024-03-25T18:29:28.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.227+0000 7fd999825640 1 -- 172.21.15.22:0/3445618837 wait complete. 2024-03-25T18:29:28.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:28.227+0000 7fd999825640 1 librados: shutdown 2024-03-25T18:29:28.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:35.047 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:35.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.147+0000 7f5790631640 10 monclient: get_monmap_and_config 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.147+0000 7f5790631640 10 monclient: build_initial_monmap 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.147+0000 7f5790631640 10 monclient: monmap: 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:35.148776+0000 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:35.148776+0000 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:35.149 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 1 Processor -- start 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 1 -- start start 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient: init 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient(hunting): picked mon.noname-b con 0x7f57880b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient(hunting): picked mon.noname-c con 0x7f57880b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient(hunting): picked mon.noname-a con 0x7f578814be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880af940 con 0x7f578814be90 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880b0a70 con 0x7f57880b1730 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880b0d50 con 0x7f57880b4d60 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:35.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.148+0000 7f5790631640 10 monclient(hunting): authenticate will time out at 7774.372434s 2024-03-25T18:29:35.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.149+0000 7f578e3cd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f57880b4d60 0x7f578813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54590/0 (socket says 172.21.15.22:54590) 2024-03-25T18:29:35.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.149+0000 7f578e3cd640 1 -- 172.21.15.22:0/3038391867 learned_addr learned my addr 172.21.15.22:0/3038391867 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:35.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.184+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195333088 0 0) 0x7f57880af940 con 0x7f578814be90 2024-03-25T18:29:35.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.184+0000 7f578d3cb640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.184+0000 7f578d3cb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.184+0000 7f578d3cb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.184+0000 7f578d3cb640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:29:35.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.184+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5774003520 con 0x7f578814be90 2024-03-25T18:29:35.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469826147 0 0) 0x7f5774003520 con 0x7f578814be90 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 >> v1:172.21.15.22:6790/0 conn(0x7f57880b4d60 legacy=0x7f578813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 >> v1:172.21.15.33:6789/0 conn(0x7f57880b1730 legacy=0x7f57880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 10 monclient: _finish_hunting 0 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 1 monclient: found mon.noname-a 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57880b1030 con 0x7f578814be90 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 10 monclient: _finish_auth 0 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 10 monclient: _check_auth_tickets 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5780001a10 con 0x7f578814be90 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:35.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f578d3cb640 10 monclient: dump: 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f5790631640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:29:35.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.203+0000 7f5790631640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:35.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5780001b70 con 0x7f578814be90 2024-03-25T18:29:35.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f578d3cb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:35.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f578d3cb640 1 -- 172.21.15.22:0/3038391867 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5780001fc0 con 0x7f578814be90 2024-03-25T18:29:35.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f578d3cb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f578d3cb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f578d3cb640 10 monclient: dump: 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f5790631640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:35.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.222+0000 7f5790631640 4 set_mon_vals no callback set 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.223+0000 7f5790631640 10 monclient: shutdown 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.223+0000 7f5790631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.223+0000 7f5790631640 1 -- 172.21.15.22:0/3038391867 >> v1:172.21.15.22:6789/0 conn(0x7f578814be90 legacy=0x7f578814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.223+0000 7f5790631640 1 -- 172.21.15.22:0/3038391867 shutdown_connections 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.223+0000 7f5790631640 1 -- 172.21.15.22:0/3038391867 >> 172.21.15.22:0/3038391867 conn(0x7f57880617a0 msgr2=0x7f5788061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.223+0000 7f5790631640 1 -- 172.21.15.22:0/3038391867 shutdown_connections 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.223+0000 7f5790631640 1 -- 172.21.15.22:0/3038391867 wait complete. 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 10 monclient: build_initial_monmap 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 10 monclient: monmap: 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:35.226 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 1 librados: starting msgr at 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 1 librados: starting objecter 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 1 Processor -- start 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 1 -- start start 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 1 librados: setting wanted keys 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 1 librados: calling monclient init 2024-03-25T18:29:35.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.224+0000 7f5790631640 10 monclient: init 2024-03-25T18:29:35.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:35.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 10 monclient(hunting): picked mon.noname-c con 0x7f57880b1730 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 10 monclient(hunting): picked mon.noname-b con 0x7f57880b4d60 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 10 monclient(hunting): picked mon.noname-a con 0x7f578814be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5788140380 con 0x7f578814be90 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5788140660 con 0x7f57880b4d60 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5788141e30 con 0x7f57880b1730 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:35.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.225+0000 7f5790631640 10 monclient(hunting): authenticate will time out at 7774.449390s 2024-03-25T18:29:35.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.236+0000 7f578ebce640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f578814be90 0x7f57881e2740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:33646/0 (socket says 172.21.15.22:33646) 2024-03-25T18:29:35.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.236+0000 7f578ebce640 1 -- 172.21.15.22:0/135829049 learned_addr learned my addr 172.21.15.22:0/135829049 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:35.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.238+0000 7f577effd640 10 client.?.objecter ms_handle_connect 0x7f578814be90 2024-03-25T18:29:35.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.238+0000 7f577effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:35.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.245+0000 7f577effd640 10 client.?.objecter ms_handle_connect 0x7f57880b1730 2024-03-25T18:29:35.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.245+0000 7f577effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.245+0000 7f577effd640 10 client.?.objecter ms_handle_connect 0x7f57880b4d60 2024-03-25T18:29:35.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.245+0000 7f577effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:35.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.254+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325051408 0 0) 0x7f5788140380 con 0x7f578814be90 2024-03-25T18:29:35.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.254+0000 7f577effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.254+0000 7f577effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.254+0000 7f577effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.254+0000 7f577effd640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:29:35.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.254+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f576c004eb0 con 0x7f578814be90 2024-03-25T18:29:35.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.259+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245319338 0 0) 0x7f5788141e30 con 0x7f57880b1730 2024-03-25T18:29:35.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.259+0000 7f577effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.259+0000 7f577effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.259+0000 7f577effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.259+0000 7f577effd640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:29:35.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.259+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5788140380 con 0x7f57880b1730 2024-03-25T18:29:35.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.271+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 615186877 0 0) 0x7f576c004eb0 con 0x7f578814be90 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.271+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5788141e30 con 0x7f578814be90 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.271+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5780001fc0 con 0x7f578814be90 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.271+0000 7f577effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.271+0000 7f577effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.271+0000 7f577effd640 10 monclient(hunting): dump: 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:35.273 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.271+0000 7f577effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.272+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116615955 0 0) 0x7f5788140660 con 0x7f57880b4d60 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.272+0000 7f577effd640 10 monclient(hunting): _negotiate 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.272+0000 7f577effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.272+0000 7f577effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:35.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.272+0000 7f577effd640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:29:35.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.272+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f576c004eb0 con 0x7f57880b4d60 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.274+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716732907 0 0) 0x7f5788140380 con 0x7f57880b1730 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.274+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5788140660 con 0x7f57880b1730 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.274+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f57840018a0 con 0x7f57880b1730 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.274+0000 7f577effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.274+0000 7f577effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.274+0000 7f577effd640 10 monclient(hunting): dump: 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.276 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:35.277 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:35.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3972226462 0 0) 0x7f5788141e30 con 0x7f578814be90 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 >> v1:172.21.15.22:6790/0 conn(0x7f57880b1730 legacy=0x7f5788138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 >> v1:172.21.15.33:6789/0 conn(0x7f57880b4d60 legacy=0x7f5788142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 10 monclient: _finish_hunting 0 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 1 monclient: found mon.a 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5788142110 con 0x7f578814be90 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 10 monclient: _finish_auth 0 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 10 monclient: _check_auth_tickets 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f577effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:35.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 5 monclient: authenticate success, global_id 4683 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 10 monclient: _renew_subs 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5788140e60 con 0x7f578814be90 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 10 monclient: _renew_subs 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5788142c40 con 0x7f578814be90 2024-03-25T18:29:35.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.280+0000 7f5790631640 1 librados: init done 2024-03-25T18:29:35.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.281+0000 7f5790631640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:35.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.281+0000 7f5790631640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:35.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.281+0000 7f5790631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:35.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.281+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57880627b0 con 0x7f578814be90 2024-03-25T18:29:35.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5780003c20 con 0x7f578814be90 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f577effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f5780003f70 con 0x7f578814be90 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f577effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f577effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f577effd640 10 monclient: dump: 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:35.611 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:35.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f578cbca640 4 set_mon_vals no callback set 2024-03-25T18:29:35.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.609+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f5780004190 con 0x7f578814be90 2024-03-25T18:29:35.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.610+0000 7f578e3cd640 10 monclient: get_auth_request con 0x7f576c06dfb0 auth_method 0 2024-03-25T18:29:35.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.610+0000 7f577effd640 10 client.4683.objecter ms_handle_connect 0x7f576c06dfb0 2024-03-25T18:29:35.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f5780054b40 con 0x7f578814be90 2024-03-25T18:29:35.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 10 client.4683.objecter ms_dispatch 0x7f5788000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:35.620 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:35.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:35.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T18:29:35.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5780050650 con 0x7f578814be90 2024-03-25T18:29:35.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:35.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.618+0000 7f577effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:35.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.873+0000 7f5790631640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.873+0000 7f5790631640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.873+0000 7f5790631640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:35.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.873+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57880b5cf0 con 0x7f578814be90 2024-03-25T18:29:35.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.916+0000 7f577effd640 1 -- 172.21.15.22:0/135829049 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f5780052580 con 0x7f578814be90 2024-03-25T18:29:35.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.916+0000 7f577effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:35.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.916+0000 7f577effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:35.918 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:35.918 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:35.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.918+0000 7f5790631640 10 librados: watch_flush enter 2024-03-25T18:29:35.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 10 librados: watch_flush exit 2024-03-25T18:29:35.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T18:29:35.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 10 client.4683.objecter successfully canceled tick 2024-03-25T18:29:35.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 >> v1:172.21.15.33:6800/215350958 conn(0x7f576c06dfb0 legacy=0x7f576c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 10 monclient: shutdown 2024-03-25T18:29:35.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:35.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 >> v1:172.21.15.22:6789/0 conn(0x7f578814be90 legacy=0x7f57881e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:35.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 shutdown_connections 2024-03-25T18:29:35.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 >> 172.21.15.22:0/135829049 conn(0x7f57880617a0 msgr2=0x7f57880b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:35.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.919+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 shutdown_connections 2024-03-25T18:29:35.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.920+0000 7f5790631640 1 -- 172.21.15.22:0/135829049 wait complete. 2024-03-25T18:29:35.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:35.920+0000 7f5790631640 1 librados: shutdown 2024-03-25T18:29:35.933 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:42.740 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.858+0000 7f2043cff640 10 monclient: get_monmap_and_config 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.858+0000 7f2043cff640 10 monclient: build_initial_monmap 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.858+0000 7f2043cff640 10 monclient: monmap: 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:42.860138+0000 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:42.860138+0000 2024-03-25T18:29:42.860 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:42.861 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:42.861 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:42.861 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:42.861 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:42.861 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.859+0000 7f2043cff640 1 Processor -- start 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.859+0000 7f2043cff640 1 -- start start 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.859+0000 7f2043cff640 10 monclient: init 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.860+0000 7f2043cff640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.860+0000 7f2043cff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.860+0000 7f2043cff640 10 monclient(hunting): picked mon.noname-a con 0x7f203c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.860+0000 7f2043cff640 10 monclient(hunting): picked mon.noname-c con 0x7f203c13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.860+0000 7f2043cff640 10 monclient(hunting): picked mon.noname-b con 0x7f203c1430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.860+0000 7f2043cff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c142520 con 0x7f203c0a98f0 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.860+0000 7f2043cff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0b28a0 con 0x7f203c13f710 2024-03-25T18:29:42.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.861+0000 7f2043cff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0b2a30 con 0x7f203c1430c0 2024-03-25T18:29:42.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.861+0000 7f2043cff640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:42.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.861+0000 7f2043cff640 10 monclient(hunting): authenticate will time out at 7782.084620s 2024-03-25T18:29:42.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.871+0000 7f204129a640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f203c13f710 0x7f203c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:53352/0 (socket says 172.21.15.22:53352) 2024-03-25T18:29:42.873 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.871+0000 7f204129a640 1 -- 172.21.15.22:0/3107206950 learned_addr learned my addr 172.21.15.22:0/3107206950 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:42.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.880+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1244168196 0 0) 0x7f203c0b28a0 con 0x7f203c13f710 2024-03-25T18:29:42.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.880+0000 7f2040a99640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.880+0000 7f2040a99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.880+0000 7f2040a99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.880+0000 7f2040a99640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:29:42.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.880+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2028003520 con 0x7f203c13f710 2024-03-25T18:29:42.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.890+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206901331 0 0) 0x7f203c0b2a30 con 0x7f203c1430c0 2024-03-25T18:29:42.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.890+0000 7f2040a99640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.890+0000 7f2040a99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.890+0000 7f2040a99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.890+0000 7f2040a99640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:29:42.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.890+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c0b28a0 con 0x7f203c1430c0 2024-03-25T18:29:42.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.902+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222553836 0 0) 0x7f203c142520 con 0x7f203c0a98f0 2024-03-25T18:29:42.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.902+0000 7f2040a99640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.902+0000 7f2040a99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.902+0000 7f2040a99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.902+0000 7f2040a99640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:29:42.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.902+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c0b2a30 con 0x7f203c0a98f0 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369928885 0 0) 0x7f2028003520 con 0x7f203c13f710 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 >> v1:172.21.15.22:6790/0 conn(0x7f203c1430c0 legacy=0x7f203c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 >> v1:172.21.15.22:6789/0 conn(0x7f203c0a98f0 legacy=0x7f203c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 10 monclient: _finish_hunting 0 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 1 monclient: found mon.noname-c 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c0b44c0 con 0x7f203c13f710 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 10 monclient: _finish_auth 0 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 10 monclient: _check_auth_tickets 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f202c001690 con 0x7f203c13f710 2024-03-25T18:29:42.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2040a99640 10 monclient: dump: 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2043cff640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:29:42.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.913+0000 7f2043cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.939+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f202c0017f0 con 0x7f203c13f710 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.939+0000 7f2040a99640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.939+0000 7f2040a99640 1 -- 172.21.15.22:0/3107206950 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f202c001c00 con 0x7f203c13f710 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.939+0000 7f2040a99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.939+0000 7f2040a99640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.939+0000 7f2040a99640 10 monclient: dump: 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.940+0000 7f2043cff640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:42.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.940+0000 7f2043cff640 4 set_mon_vals no callback set 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.940+0000 7f2043cff640 10 monclient: shutdown 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.940+0000 7f2043cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.940+0000 7f2043cff640 1 -- 172.21.15.22:0/3107206950 >> v1:172.21.15.33:6789/0 conn(0x7f203c13f710 legacy=0x7f203c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.941+0000 7f2043cff640 1 -- 172.21.15.22:0/3107206950 shutdown_connections 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.941+0000 7f2043cff640 1 -- 172.21.15.22:0/3107206950 >> 172.21.15.22:0/3107206950 conn(0x7f203c0b7ea0 msgr2=0x7f203c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.941+0000 7f2043cff640 1 -- 172.21.15.22:0/3107206950 shutdown_connections 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.941+0000 7f2043cff640 1 -- 172.21.15.22:0/3107206950 wait complete. 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.942+0000 7f2043cff640 10 monclient: build_initial_monmap 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.942+0000 7f2043cff640 10 monclient: monmap: 2024-03-25T18:29:42.944 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:42.945 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.942+0000 7f2043cff640 1 librados: starting msgr at 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.942+0000 7f2043cff640 1 librados: starting objecter 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.942+0000 7f2043cff640 1 Processor -- start 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 1 -- start start 2024-03-25T18:29:42.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 1 librados: setting wanted keys 2024-03-25T18:29:42.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 1 librados: calling monclient init 2024-03-25T18:29:42.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 10 monclient: init 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 10 monclient(hunting): picked mon.noname-a con 0x7f203c0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 10 monclient(hunting): picked mon.noname-c con 0x7f203c13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.943+0000 7f2043cff640 10 monclient(hunting): picked mon.noname-b con 0x7f203c1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.944+0000 7f2043cff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0abdc0 con 0x7f203c0a98f0 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.944+0000 7f2043cff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c1d7730 con 0x7f203c1430c0 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.944+0000 7f2043cff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c138f10 con 0x7f203c13f710 2024-03-25T18:29:42.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.944+0000 7f2043cff640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:42.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.944+0000 7f2043cff640 10 monclient(hunting): authenticate will time out at 7782.167921s 2024-03-25T18:29:42.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.949+0000 7f204129a640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f203c13f710 0x7f203c1d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59390/0 (socket says 172.21.15.22:59390) 2024-03-25T18:29:42.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.949+0000 7f204129a640 1 -- 172.21.15.22:0/2726741504 learned_addr learned my addr 172.21.15.22:0/2726741504 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:42.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.952+0000 7f20327fc640 10 client.?.objecter ms_handle_connect 0x7f203c13f710 2024-03-25T18:29:42.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.952+0000 7f20327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:42.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.959+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903545252 0 0) 0x7f203c138f10 con 0x7f203c13f710 2024-03-25T18:29:42.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.959+0000 7f20327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.959+0000 7f20327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.959+0000 7f20327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.959+0000 7f20327fc640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:29:42.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.959+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2020004eb0 con 0x7f203c13f710 2024-03-25T18:29:42.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.963+0000 7f20327fc640 10 client.?.objecter ms_handle_connect 0x7f203c1430c0 2024-03-25T18:29:42.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.963+0000 7f20327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:42.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.967+0000 7f20327fc640 10 client.?.objecter ms_handle_connect 0x7f203c0a98f0 2024-03-25T18:29:42.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.967+0000 7f20327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:42.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668774512 0 0) 0x7f203c1d7730 con 0x7f203c1430c0 2024-03-25T18:29:42.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:29:42.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c138f10 con 0x7f203c1430c0 2024-03-25T18:29:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056443038 0 0) 0x7f203c0abdc0 con 0x7f203c0a98f0 2024-03-25T18:29:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:29:42.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.975+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f203c1d7730 con 0x7f203c0a98f0 2024-03-25T18:29:42.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.992+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2569115902 0 0) 0x7f203c138f10 con 0x7f203c1430c0 2024-03-25T18:29:42.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.992+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f203c0abdc0 con 0x7f203c1430c0 2024-03-25T18:29:42.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.992+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2034001b50 con 0x7f203c1430c0 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.992+0000 7f20327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.992+0000 7f20327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.992+0000 7f20327fc640 10 monclient(hunting): dump: 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.992+0000 7f20327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:42.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.993+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 458981130 0 0) 0x7f2020004eb0 con 0x7f203c13f710 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.993+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f203c138f10 con 0x7f203c13f710 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.993+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f202c001bc0 con 0x7f203c13f710 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.993+0000 7f20327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.993+0000 7f20327fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.993+0000 7f20327fc640 10 monclient(hunting): dump: 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:42.996 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.994+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 247045286 0 0) 0x7f203c1d7730 con 0x7f203c0a98f0 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.994+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2020004eb0 con 0x7f203c0a98f0 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.994+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2038001830 con 0x7f203c0a98f0 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.994+0000 7f20327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.994+0000 7f20327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:42.994+0000 7f20327fc640 10 monclient(hunting): dump: 2024-03-25T18:29:42.997 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:42.998 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:43.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.000+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 734313773 0 0) 0x7f203c138f10 con 0x7f203c13f710 2024-03-25T18:29:43.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.000+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 >> v1:172.21.15.33:6789/0 conn(0x7f203c1430c0 legacy=0x7f203c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.000+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 >> v1:172.21.15.22:6789/0 conn(0x7f203c0a98f0 legacy=0x7f203c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.000+0000 7f20327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.000+0000 7f20327fc640 1 monclient: found mon.c 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.000+0000 7f20327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f20327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c1390a0 con 0x7f203c13f710 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f20327fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f20327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f20327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 5 monclient: authenticate success, global_id 4706 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 10 monclient: _renew_subs 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f203c1d8320 con 0x7f203c13f710 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:43.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 10 monclient: _renew_subs 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c1d43c0 con 0x7f203c13f710 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.001+0000 7f2043cff640 1 librados: init done 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.002+0000 7f2043cff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.002+0000 7f2043cff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.002+0000 7f2043cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:43.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.002+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c0627b0 con 0x7f203c13f710 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.349+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f202c0011f0 con 0x7f203c13f710 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.349+0000 7f20327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.349+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f202c004c30 con 0x7f203c13f710 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.349+0000 7f20327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.349+0000 7f20327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.349+0000 7f20327fc640 10 monclient: dump: 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:43.351 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.349+0000 7f2033fff640 4 set_mon_vals no callback set 2024-03-25T18:29:43.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.350+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f202c01ac80 con 0x7f203c13f710 2024-03-25T18:29:43.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.351+0000 7f2041a9b640 10 monclient: get_auth_request con 0x7f202006e110 auth_method 0 2024-03-25T18:29:43.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.351+0000 7f20327fc640 10 client.4706.objecter ms_handle_connect 0x7f202006e110 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.355+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f202c054970 con 0x7f203c13f710 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.355+0000 7f20327fc640 10 client.4706.objecter ms_dispatch 0x7f203c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.355+0000 7f20327fc640 3 client.4706.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.355+0000 7f20327fc640 3 client.4706.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.355+0000 7f20327fc640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.356+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f202c0502b0 con 0x7f203c13f710 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.356+0000 7f20327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:43.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.356+0000 7f20327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.579+0000 7f2043cff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.579+0000 7f2043cff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.579+0000 7f2043cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:29:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.579+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f203c1d84b0 con 0x7f203c13f710 2024-03-25T18:29:43.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.649+0000 7f20327fc640 1 -- 172.21.15.22:0/2726741504 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f202c0523b0 con 0x7f203c13f710 2024-03-25T18:29:43.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.649+0000 7f20327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:43.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.649+0000 7f20327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:43.652 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:43.652 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:43.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.651+0000 7f2043cff640 10 librados: watch_flush enter 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 10 librados: watch_flush exit 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 10 client.4706.objecter successfully canceled tick 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 >> v1:172.21.15.33:6800/215350958 conn(0x7f202006e110 legacy=0x7f20200705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 10 monclient: shutdown 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 >> v1:172.21.15.22:6790/0 conn(0x7f203c13f710 legacy=0x7f203c1d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 shutdown_connections 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 >> 172.21.15.22:0/2726741504 conn(0x7f203c0b7ea0 msgr2=0x7f203c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:43.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 shutdown_connections 2024-03-25T18:29:43.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 1 -- 172.21.15.22:0/2726741504 wait complete. 2024-03-25T18:29:43.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:43.652+0000 7f2043cff640 1 librados: shutdown 2024-03-25T18:29:43.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:44.192 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:46.854 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.248 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:46.910 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.282 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:10:46.944 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.303 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:46.965 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.305 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:46.923 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.309 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:46.927 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.309 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:46.971 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.341 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:10:47.003 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:44.342 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:10:46.960 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:29:50.470 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:50.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.583+0000 7f24f94ff640 10 monclient: get_monmap_and_config 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.583+0000 7f24f94ff640 10 monclient: build_initial_monmap 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.583+0000 7f24f94ff640 10 monclient: monmap: 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:50.584962+0000 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:50.584962+0000 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:50.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.584+0000 7f24f94ff640 1 Processor -- start 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.584+0000 7f24f94ff640 1 -- start start 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.584+0000 7f24f94ff640 10 monclient: init 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 10 monclient(hunting): picked mon.noname-c con 0x7f24f40a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 10 monclient(hunting): picked mon.noname-a con 0x7f24f413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 10 monclient(hunting): picked mon.noname-b con 0x7f24f41430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f4142520 con 0x7f24f413f710 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f40b28a0 con 0x7f24f40a98f0 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f40b2a30 con 0x7f24f41430c0 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:50.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.585+0000 7f24f94ff640 10 monclient(hunting): authenticate will time out at 7789.808774s 2024-03-25T18:29:50.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.597+0000 7f24f37fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f24f41430c0 0x7f24f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34916/0 (socket says 172.21.15.22:34916) 2024-03-25T18:29:50.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.597+0000 7f24f37fe640 1 -- 172.21.15.22:0/42512405 learned_addr learned my addr 172.21.15.22:0/42512405 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:50.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.607+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526387391 0 0) 0x7f24f40b28a0 con 0x7f24f40a98f0 2024-03-25T18:29:50.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.607+0000 7f24f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.607+0000 7f24f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.607+0000 7f24f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.607+0000 7f24f27fc640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:29:50.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.607+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24e0003520 con 0x7f24f40a98f0 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2603871826 0 0) 0x7f24e0003520 con 0x7f24f40a98f0 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 >> v1:172.21.15.22:6790/0 conn(0x7f24f41430c0 legacy=0x7f24f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 >> v1:172.21.15.22:6789/0 conn(0x7f24f413f710 legacy=0x7f24f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f40b44c0 con 0x7f24f40a98f0 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f24e4001690 con 0x7f24f40a98f0 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f27fc640 10 monclient: dump: 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f94ff640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:29:50.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.615+0000 7f24f94ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:50.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24e40017f0 con 0x7f24f40a98f0 2024-03-25T18:29:50.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:50.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f27fc640 1 -- 172.21.15.22:0/42512405 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f24e4001c00 con 0x7f24f40a98f0 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f94ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f94ff640 4 set_mon_vals no callback set 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f27fc640 10 monclient: dump: 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:50.626 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:50.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f94ff640 10 monclient: shutdown 2024-03-25T18:29:50.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f94ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:50.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.624+0000 7f24f94ff640 1 -- 172.21.15.22:0/42512405 >> v1:172.21.15.33:6789/0 conn(0x7f24f40a98f0 legacy=0x7f24f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.625+0000 7f24f94ff640 1 -- 172.21.15.22:0/42512405 shutdown_connections 2024-03-25T18:29:50.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.625+0000 7f24f94ff640 1 -- 172.21.15.22:0/42512405 >> 172.21.15.22:0/42512405 conn(0x7f24f40b7ea0 msgr2=0x7f24f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:50.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.625+0000 7f24f94ff640 1 -- 172.21.15.22:0/42512405 shutdown_connections 2024-03-25T18:29:50.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.625+0000 7f24f94ff640 1 -- 172.21.15.22:0/42512405 wait complete. 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.625+0000 7f24f94ff640 10 monclient: build_initial_monmap 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.625+0000 7f24f94ff640 10 monclient: monmap: 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 librados: starting msgr at 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 librados: starting objecter 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 Processor -- start 2024-03-25T18:29:50.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 -- start start 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 librados: setting wanted keys 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 librados: calling monclient init 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient: init 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient(hunting): picked mon.noname-c con 0x7f24f40a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient(hunting): picked mon.noname-a con 0x7f24f413f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient(hunting): picked mon.noname-b con 0x7f24f41430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f40ab9e0 con 0x7f24f413f710 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f4138c20 con 0x7f24f41430c0 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f4138f00 con 0x7f24f40a98f0 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:50.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.626+0000 7f24f94ff640 10 monclient(hunting): authenticate will time out at 7789.850496s 2024-03-25T18:29:50.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.634+0000 7f24f2ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f24f40a98f0 0x7f24f4137fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34950/0 (socket says 172.21.15.22:34950) 2024-03-25T18:29:50.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.634+0000 7f24f2ffd640 1 -- 172.21.15.22:0/4077719757 learned_addr learned my addr 172.21.15.22:0/4077719757 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:50.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.636+0000 7f24eb7fe640 10 client.?.objecter ms_handle_connect 0x7f24f40a98f0 2024-03-25T18:29:50.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.636+0000 7f24eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:50.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.641+0000 7f24eb7fe640 10 client.?.objecter ms_handle_connect 0x7f24f41430c0 2024-03-25T18:29:50.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.641+0000 7f24eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:50.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.649+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119927174 0 0) 0x7f24f4138c20 con 0x7f24f41430c0 2024-03-25T18:29:50.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.650+0000 7f24eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.650+0000 7f24eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.650+0000 7f24eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.650+0000 7f24eb7fe640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:29:50.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.650+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4004eb0 con 0x7f24f41430c0 2024-03-25T18:29:50.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.650+0000 7f24eb7fe640 10 client.?.objecter ms_handle_connect 0x7f24f413f710 2024-03-25T18:29:50.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.650+0000 7f24eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:50.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.651+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042265561 0 0) 0x7f24f4138f00 con 0x7f24f40a98f0 2024-03-25T18:29:50.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.651+0000 7f24eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.651+0000 7f24eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.651+0000 7f24eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.651+0000 7f24eb7fe640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:29:50.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.651+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f4138c20 con 0x7f24f40a98f0 2024-03-25T18:29:50.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.671+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285333625 0 0) 0x7f24f40ab9e0 con 0x7f24f413f710 2024-03-25T18:29:50.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.671+0000 7f24eb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:29:50.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.671+0000 7f24eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:50.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.671+0000 7f24eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:50.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.671+0000 7f24eb7fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:29:50.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.671+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f4138f00 con 0x7f24f413f710 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224229574 0 0) 0x7f24d4004eb0 con 0x7f24f41430c0 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24f40ab9e0 con 0x7f24f41430c0 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f24ec001b00 con 0x7f24f41430c0 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 10 monclient(hunting): dump: 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.688 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897855225 0 0) 0x7f24f4138f00 con 0x7f24f413f710 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24d4004eb0 con 0x7f24f413f710 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f24dc001830 con 0x7f24f413f710 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.686+0000 7f24eb7fe640 10 monclient(hunting): dump: 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:50.689 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.690 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.690 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.690 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:50.690 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:50.690 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:50.690 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:50.690 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:50.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.696+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1733314849 0 0) 0x7f24f4138c20 con 0x7f24f40a98f0 2024-03-25T18:29:50.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.696+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24f4138f00 con 0x7f24f40a98f0 2024-03-25T18:29:50.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.696+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f24e4001bc0 con 0x7f24f40a98f0 2024-03-25T18:29:50.697 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.696+0000 7f24eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.696+0000 7f24eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.696+0000 7f24eb7fe640 10 monclient(hunting): dump: 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:50.698 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:50.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 187935064 0 0) 0x7f24d4004eb0 con 0x7f24f413f710 2024-03-25T18:29:50.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 >> v1:172.21.15.22:6790/0 conn(0x7f24f40a98f0 legacy=0x7f24f4137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 >> v1:172.21.15.33:6789/0 conn(0x7f24f41430c0 legacy=0x7f24f41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:50.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:29:50.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 1 monclient: found mon.a 2024-03-25T18:29:50.704 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f414e7a0 con 0x7f24f413f710 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 10 monclient: _renew_subs 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f41386d0 con 0x7f24f413f710 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 10 monclient: _renew_subs 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f41d8060 con 0x7f24f413f710 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.702+0000 7f24f94ff640 1 librados: init done 2024-03-25T18:29:50.705 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.703+0000 7f24f94ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:50.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.703+0000 7f24f94ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:50.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.703+0000 7f24f94ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:50.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:50.703+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f4074cc0 con 0x7f24f413f710 2024-03-25T18:29:51.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.041+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24dc001d40 con 0x7f24f413f710 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.041+0000 7f24eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.041+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f24dc003790 con 0x7f24f413f710 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.041+0000 7f24eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.041+0000 7f24eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.041+0000 7f24eb7fe640 10 monclient: dump: 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:51.043 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:51.044 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:51.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.041+0000 7f24f1ffb640 4 set_mon_vals no callback set 2024-03-25T18:29:51.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.042+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f24dc0039b0 con 0x7f24f413f710 2024-03-25T18:29:51.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.043+0000 7f24f2ffd640 10 monclient: get_auth_request con 0x7f24d406e160 auth_method 0 2024-03-25T18:29:51.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.043+0000 7f24eb7fe640 10 client.4695.objecter ms_handle_connect 0x7f24d406e160 2024-03-25T18:29:51.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.047+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f24dc053590 con 0x7f24f413f710 2024-03-25T18:29:51.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.047+0000 7f24eb7fe640 10 client.4695.objecter ms_dispatch 0x7f24f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:51.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.047+0000 7f24eb7fe640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:51.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.047+0000 7f24eb7fe640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:51.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.048+0000 7f24eb7fe640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T18:29:51.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.048+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24dc04eed0 con 0x7f24f413f710 2024-03-25T18:29:51.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.048+0000 7f24eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:51.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.048+0000 7f24eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:51.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.330+0000 7f24f94ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:51.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.330+0000 7f24f94ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:51.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.330+0000 7f24f94ff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-25T18:29:51.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.330+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24f40627b0 con 0x7f24f413f710 2024-03-25T18:29:51.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.368+0000 7f24eb7fe640 1 -- 172.21.15.22:0/4077719757 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f24dc050fd0 con 0x7f24f413f710 2024-03-25T18:29:51.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.368+0000 7f24eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:51.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.368+0000 7f24eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:51.370 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:51.370 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 10 librados: watch_flush enter 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 10 librados: watch_flush exit 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 10 client.4695.objecter successfully canceled tick 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 >> v1:172.21.15.33:6800/215350958 conn(0x7f24d406e160 legacy=0x7f24d4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 10 monclient: shutdown 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 >> v1:172.21.15.22:6789/0 conn(0x7f24f413f710 legacy=0x7f24f41d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 shutdown_connections 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 >> 172.21.15.22:0/4077719757 conn(0x7f24f40b7ea0 msgr2=0x7f24f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:51.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 shutdown_connections 2024-03-25T18:29:51.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 1 -- 172.21.15.22:0/4077719757 wait complete. 2024-03-25T18:29:51.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:51.370+0000 7f24f94ff640 1 librados: shutdown 2024-03-25T18:29:51.384 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:29:58.191 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.310+0000 7fafd584a640 10 monclient: get_monmap_and_config 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.310+0000 7fafd584a640 10 monclient: build_initial_monmap 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.310+0000 7fafd584a640 10 monclient: monmap: 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:29:58.312207+0000 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:29:58.312207+0000 2024-03-25T18:29:58.312 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:58.313 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:58.313 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:58.313 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:29:58.313 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:29:58.313 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.311+0000 7fafd584a640 1 Processor -- start 2024-03-25T18:29:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.311+0000 7fafd584a640 1 -- start start 2024-03-25T18:29:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.311+0000 7fafd584a640 10 monclient: init 2024-03-25T18:29:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.312+0000 7fafd584a640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.312+0000 7fafd584a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:29:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.312+0000 7fafd584a640 10 monclient(hunting): picked mon.noname-c con 0x7fafd00adaf0 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.312+0000 7fafd584a640 10 monclient(hunting): picked mon.noname-b con 0x7fafd013fd90 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.312+0000 7fafd584a640 10 monclient(hunting): picked mon.noname-a con 0x7fafd00afcb0 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.313+0000 7fafd584a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd00b00e0 con 0x7fafd00afcb0 2024-03-25T18:29:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.313+0000 7fafd584a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd006fc30 con 0x7fafd00adaf0 2024-03-25T18:29:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.313+0000 7fafd584a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd006e510 con 0x7fafd013fd90 2024-03-25T18:29:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.313+0000 7fafd584a640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.313+0000 7fafd584a640 10 monclient(hunting): authenticate will time out at 7797.536797s 2024-03-25T18:29:58.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.318+0000 7fafceffd640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fafd00adaf0 0x7fafd013f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:53804/0 (socket says 172.21.15.22:53804) 2024-03-25T18:29:58.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.318+0000 7fafceffd640 1 -- 172.21.15.22:0/2830460755 learned_addr learned my addr 172.21.15.22:0/2830460755 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.329+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80639688 0 0) 0x7fafd006fc30 con 0x7fafd00adaf0 2024-03-25T18:29:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.330+0000 7fafce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.330+0000 7fafce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.330+0000 7fafce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.330+0000 7fafce7fc640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:29:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.330+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafbc003520 con 0x7fafd00adaf0 2024-03-25T18:29:58.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.339+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971607377 0 0) 0x7fafd006e510 con 0x7fafd013fd90 2024-03-25T18:29:58.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.339+0000 7fafce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.339+0000 7fafce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.339+0000 7fafce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.339+0000 7fafce7fc640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:29:58.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.339+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd006fc30 con 0x7fafd013fd90 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1669830987 0 0) 0x7fafbc003520 con 0x7fafd00adaf0 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 >> v1:172.21.15.22:6790/0 conn(0x7fafd013fd90 legacy=0x7fafd0142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 >> v1:172.21.15.22:6789/0 conn(0x7fafd00afcb0 legacy=0x7fafd014dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 1 monclient: found mon.noname-c 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd00b0340 con 0x7fafd00adaf0 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 10 monclient: _finish_auth 0 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fafc0001690 con 0x7fafd00adaf0 2024-03-25T18:29:58.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafce7fc640 10 monclient: dump: 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafd584a640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:29:58.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.347+0000 7fafd584a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafc00017f0 con 0x7fafd00adaf0 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafce7fc640 1 -- 172.21.15.22:0/2830460755 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fafc0001c00 con 0x7fafd00adaf0 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafce7fc640 10 monclient: dump: 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:58.364 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:58.365 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:58.365 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:58.365 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:58.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafd584a640 10 monclient: get_monmap_and_config success 2024-03-25T18:29:58.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.362+0000 7fafd584a640 4 set_mon_vals no callback set 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.363+0000 7fafd584a640 10 monclient: shutdown 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.363+0000 7fafd584a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.363+0000 7fafd584a640 1 -- 172.21.15.22:0/2830460755 >> v1:172.21.15.33:6789/0 conn(0x7fafd00adaf0 legacy=0x7fafd013f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.363+0000 7fafd584a640 1 -- 172.21.15.22:0/2830460755 shutdown_connections 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.363+0000 7fafd584a640 1 -- 172.21.15.22:0/2830460755 >> 172.21.15.22:0/2830460755 conn(0x7fafd00b3e60 msgr2=0x7fafd00b4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.363+0000 7fafd584a640 1 -- 172.21.15.22:0/2830460755 shutdown_connections 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.363+0000 7fafd584a640 1 -- 172.21.15.22:0/2830460755 wait complete. 2024-03-25T18:29:58.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 10 monclient: build_initial_monmap 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 10 monclient: monmap: 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 1 librados: starting msgr at 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 1 librados: starting objecter 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 1 Processor -- start 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 1 -- start start 2024-03-25T18:29:58.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 1 librados: setting wanted keys 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 1 librados: calling monclient init 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.364+0000 7fafd584a640 10 monclient: init 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 10 monclient: _reopen_session rank -1 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 10 monclient(hunting): picked mon.noname-c con 0x7fafd00adaf0 addr v1:172.21.15.22:6790/0 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 10 monclient(hunting): picked mon.noname-a con 0x7fafd00afcb0 addr v1:172.21.15.22:6789/0 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 10 monclient(hunting): picked mon.noname-b con 0x7fafd013fd90 addr v1:172.21.15.33:6789/0 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd01441c0 con 0x7fafd00afcb0 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd0146440 con 0x7fafd013fd90 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd0143950 con 0x7fafd00adaf0 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 10 monclient(hunting): _renew_subs 2024-03-25T18:29:58.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.365+0000 7fafd584a640 10 monclient(hunting): authenticate will time out at 7797.589127s 2024-03-25T18:29:58.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.377+0000 7fafcf7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7fafd013fd90 0x7fafd01da800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:53838/0 (socket says 172.21.15.22:53838) 2024-03-25T18:29:58.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.377+0000 7fafcf7fe640 1 -- 172.21.15.22:0/3596837846 learned_addr learned my addr 172.21.15.22:0/3596837846 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:29:58.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.382+0000 7fafccff9640 10 client.?.objecter ms_handle_connect 0x7fafd013fd90 2024-03-25T18:29:58.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.382+0000 7fafccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.385+0000 7fafccff9640 10 client.?.objecter ms_handle_connect 0x7fafd00afcb0 2024-03-25T18:29:58.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.385+0000 7fafccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:58.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 10 client.?.objecter ms_handle_connect 0x7fafd00adaf0 2024-03-25T18:29:58.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:29:58.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890674345 0 0) 0x7fafd0146440 con 0x7fafd013fd90 2024-03-25T18:29:58.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:29:58.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.390+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb0004eb0 con 0x7fafd013fd90 2024-03-25T18:29:58.401 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.400+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957650693 0 0) 0x7fafd0143950 con 0x7fafd00adaf0 2024-03-25T18:29:58.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.400+0000 7fafccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.400+0000 7fafccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.400+0000 7fafccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.400+0000 7fafccff9640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:29:58.402 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.400+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd0146440 con 0x7fafd00adaf0 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.407+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3232963606 0 0) 0x7fafb0004eb0 con 0x7fafd013fd90 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.407+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafd0143950 con 0x7fafd013fd90 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.407+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fafc8001d30 con 0x7fafd013fd90 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.407+0000 7fafccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.407+0000 7fafccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.407+0000 7fafccff9640 10 monclient(hunting): dump: 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:58.409 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:58.410 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:58.410 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:58.410 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:58.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.407+0000 7fafccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:29:58.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.409+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388012168 0 0) 0x7fafd01441c0 con 0x7fafd00afcb0 2024-03-25T18:29:58.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.409+0000 7fafccff9640 10 monclient(hunting): _negotiate 2024-03-25T18:29:58.410 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.409+0000 7fafccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:29:58.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.409+0000 7fafccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:29:58.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.409+0000 7fafccff9640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:29:58.411 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.409+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb0004eb0 con 0x7fafd00afcb0 2024-03-25T18:29:58.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3543946230 0 0) 0x7fafd0143950 con 0x7fafd013fd90 2024-03-25T18:29:58.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 >> v1:172.21.15.22:6790/0 conn(0x7fafd00adaf0 legacy=0x7fafd0137e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 >> v1:172.21.15.22:6789/0 conn(0x7fafd00afcb0 legacy=0x7fafd0144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:58.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 10 monclient: _finish_hunting 0 2024-03-25T18:29:58.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 1 monclient: found mon.b 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd0138920 con 0x7fafd013fd90 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 10 monclient: _finish_auth 0 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 10 monclient: _check_auth_tickets 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 10 monclient: _renew_subs 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd0138540 con 0x7fafd013fd90 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 10 monclient: _renew_subs 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd0144e80 con 0x7fafd013fd90 2024-03-25T18:29:58.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.422+0000 7fafd584a640 1 librados: init done 2024-03-25T18:29:58.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.423+0000 7fafd584a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:58.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.423+0000 7fafd584a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:58.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.423+0000 7fafd584a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:58.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.423+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd0144390 con 0x7fafd013fd90 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.715+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafc8003990 con 0x7fafd013fd90 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.715+0000 7fafccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.715+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7fafc8003ca0 con 0x7fafd013fd90 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.715+0000 7fafccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.715+0000 7fafccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.715+0000 7fafccff9640 10 monclient: dump: 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:29:58.717 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:29:58.718 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:29:58.718 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:29:58.718 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:29:58.718 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:29:58.718 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:29:58.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.715+0000 7fafcdffb640 4 set_mon_vals no callback set 2024-03-25T18:29:58.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.716+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7fafc8003e80 con 0x7fafd013fd90 2024-03-25T18:29:58.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.717+0000 7fafceffd640 10 monclient: get_auth_request con 0x7fafb006de00 auth_method 0 2024-03-25T18:29:58.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.717+0000 7fafccff9640 10 client.4714.objecter ms_handle_connect 0x7fafb006de00 2024-03-25T18:29:58.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.728+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7fafc8054bc0 con 0x7fafd013fd90 2024-03-25T18:29:58.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.728+0000 7fafccff9640 10 client.4714.objecter ms_dispatch 0x7fafd0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:29:58.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.728+0000 7fafccff9640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:29:58.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.728+0000 7fafccff9640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:29:58.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.728+0000 7fafccff9640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T18:29:58.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.729+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafc80506d0 con 0x7fafd013fd90 2024-03-25T18:29:58.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.729+0000 7fafccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:29:58.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.729+0000 7fafccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:29:58.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.957+0000 7fafd584a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.957+0000 7fafd584a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:58.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.957+0000 7fafd584a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:29:58.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.957+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafd0138ab0 con 0x7fafd013fd90 2024-03-25T18:29:58.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.998+0000 7fafccff9640 1 -- 172.21.15.22:0/3596837846 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7fafc8052600 con 0x7fafd013fd90 2024-03-25T18:29:59.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.998+0000 7fafccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:29:59.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:58.998+0000 7fafccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:29:59.000 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:29:59.000 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:29:59.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.000+0000 7fafd584a640 10 librados: watch_flush enter 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.000+0000 7fafd584a640 10 librados: watch_flush exit 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.000+0000 7fafd584a640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.000+0000 7fafd584a640 10 client.4714.objecter successfully canceled tick 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 >> v1:172.21.15.33:6800/215350958 conn(0x7fafb006de00 legacy=0x7fafb00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 10 monclient: shutdown 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 >> v1:172.21.15.33:6789/0 conn(0x7fafd013fd90 legacy=0x7fafd01da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 shutdown_connections 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 >> 172.21.15.22:0/3596837846 conn(0x7fafd00b3e60 msgr2=0x7fafd013f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 shutdown_connections 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 1 -- 172.21.15.22:0/3596837846 wait complete. 2024-03-25T18:29:59.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:29:59.001+0000 7fafd584a640 1 librados: shutdown 2024-03-25T18:29:59.015 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:04.365 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:07.026 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.375 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:07.037 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.415 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:07.077 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.417 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:07.079 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.425 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:07.087 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.430 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:07.048 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.437 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:07.055 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.456 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:07.074 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:04.457 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:07.118 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:05.821 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:05.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.940+0000 7f7e0ed9d640 10 monclient: get_monmap_and_config 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.940+0000 7f7e0ed9d640 10 monclient: build_initial_monmap 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.940+0000 7f7e0ed9d640 10 monclient: monmap: 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:30:05.941643+0000 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:30:05.941643+0000 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:30:05.942 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 1 Processor -- start 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 1 -- start start 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 10 monclient: init 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e080a98f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e0813f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e081430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e08142520 con 0x7f7e0813f710 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e080b28a0 con 0x7f7e080a98f0 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e080b2a30 con 0x7f7e081430c0 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.941+0000 7f7e0ed9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:05.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.942+0000 7f7e0ed9d640 10 monclient(hunting): authenticate will time out at 7805.165582s 2024-03-25T18:30:05.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.947+0000 7f7dfffff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f7e0813f710 0x7f7e081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52860/0 (socket says 172.21.15.22:52860) 2024-03-25T18:30:05.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.947+0000 7f7dfffff640 1 -- 172.21.15.22:0/4150134587 learned_addr learned my addr 172.21.15.22:0/4150134587 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:05.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.954+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821770741 0 0) 0x7f7e08142520 con 0x7f7e0813f710 2024-03-25T18:30:05.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.954+0000 7f7dff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.954+0000 7f7dff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.954+0000 7f7dff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.954+0000 7f7dff7fe640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:30:05.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.954+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df0003520 con 0x7f7e0813f710 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.964+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983280663 0 0) 0x7f7e080b28a0 con 0x7f7e080a98f0 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.964+0000 7f7dff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.964+0000 7f7dff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.964+0000 7f7dff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.964+0000 7f7dff7fe640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:30:05.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.964+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e08142520 con 0x7f7e080a98f0 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3493150019 0 0) 0x7f7df0003520 con 0x7f7e0813f710 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 >> v1:172.21.15.22:6790/0 conn(0x7f7e081430c0 legacy=0x7f7e0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 >> v1:172.21.15.33:6789/0 conn(0x7f7e080a98f0 legacy=0x7f7e080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 1 monclient: found mon.noname-a 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e080b44c0 con 0x7f7e0813f710 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:05.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7e0ed9d640 5 monclient: authenticate success, global_id 4704 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7df8001690 con 0x7f7e0813f710 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7e0ed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7dff7fe640 10 monclient: dump: 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:05.969 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:05.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.966+0000 7f7e0ed9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:05.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7df80017f0 con 0x7f7e0813f710 2024-03-25T18:30:05.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7dff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:05.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7dff7fe640 1 -- 172.21.15.22:0/4150134587 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7df8001c00 con 0x7f7e0813f710 2024-03-25T18:30:05.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7dff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7dff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7dff7fe640 10 monclient: dump: 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7e0ed9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:05.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7e0ed9d640 4 set_mon_vals no callback set 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7e0ed9d640 10 monclient: shutdown 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7e0ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.976+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/4150134587 >> v1:172.21.15.22:6789/0 conn(0x7f7e0813f710 legacy=0x7f7e081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.977+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/4150134587 shutdown_connections 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.977+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/4150134587 >> 172.21.15.22:0/4150134587 conn(0x7f7e080b7ea0 msgr2=0x7f7e080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.977+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/4150134587 shutdown_connections 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.977+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/4150134587 wait complete. 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 10 monclient: build_initial_monmap 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 10 monclient: monmap: 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:30:05.980 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 1 librados: starting msgr at 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 1 librados: starting objecter 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 1 Processor -- start 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 1 -- start start 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 1 librados: setting wanted keys 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 1 librados: calling monclient init 2024-03-25T18:30:05.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.978+0000 7f7e0ed9d640 10 monclient: init 2024-03-25T18:30:05.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e080a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e0813f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e081430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e080abd70 con 0x7f7e081430c0 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e081d7710 con 0x7f7e0813f710 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e081d8340 con 0x7f7e080a98f0 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:05.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.979+0000 7f7e0ed9d640 10 monclient(hunting): authenticate will time out at 7805.203182s 2024-03-25T18:30:05.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.992+0000 7f7dfffff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f7e0813f710 0x7f7e081d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:45720/0 (socket says 172.21.15.22:45720) 2024-03-25T18:30:05.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.992+0000 7f7dfffff640 1 -- 172.21.15.22:0/3068338228 learned_addr learned my addr 172.21.15.22:0/3068338228 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.995+0000 7f7dfd7fa640 10 client.?.objecter ms_handle_connect 0x7f7e0813f710 2024-03-25T18:30:05.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.995+0000 7f7dfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.998+0000 7f7dfd7fa640 10 client.?.objecter ms_handle_connect 0x7f7e081430c0 2024-03-25T18:30:05.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.998+0000 7f7dfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.999+0000 7f7dfd7fa640 10 client.?.objecter ms_handle_connect 0x7f7e080a98f0 2024-03-25T18:30:06.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:05.999+0000 7f7dfd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.004+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428050568 0 0) 0x7f7e081d7710 con 0x7f7e0813f710 2024-03-25T18:30:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.004+0000 7f7dfd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.004+0000 7f7dfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.004+0000 7f7dfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.004+0000 7f7dfd7fa640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:30:06.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.004+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7df4004eb0 con 0x7f7e0813f710 2024-03-25T18:30:06.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.005+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762825294 0 0) 0x7f7e080abd70 con 0x7f7e081430c0 2024-03-25T18:30:06.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.005+0000 7f7dfd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:06.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.005+0000 7f7dfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:06.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.005+0000 7f7dfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:06.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.005+0000 7f7dfd7fa640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:30:06.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.005+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e081d7710 con 0x7f7e081430c0 2024-03-25T18:30:06.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.008+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187550568 0 0) 0x7f7e081d8340 con 0x7f7e080a98f0 2024-03-25T18:30:06.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.008+0000 7f7dfd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:30:06.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.008+0000 7f7dfd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:06.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.008+0000 7f7dfd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:06.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.008+0000 7f7dfd7fa640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:30:06.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.008+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e080abd70 con 0x7f7e080a98f0 2024-03-25T18:30:06.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.019+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1249229823 0 0) 0x7f7e081d7710 con 0x7f7e081430c0 2024-03-25T18:30:06.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.019+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e081d8340 con 0x7f7e081430c0 2024-03-25T18:30:06.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.019+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7e00001b00 con 0x7f7e081430c0 2024-03-25T18:30:06.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.019+0000 7f7dfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.019+0000 7f7dfd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.019+0000 7f7dfd7fa640 10 monclient(hunting): dump: 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:06.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.019+0000 7f7dfd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:06.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.025+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1434288392 0 0) 0x7f7df4004eb0 con 0x7f7e0813f710 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.025+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e081d7710 con 0x7f7e0813f710 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.025+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7df8001bc0 con 0x7f7e0813f710 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.025+0000 7f7dfd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.025+0000 7f7dfd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.025+0000 7f7dfd7fa640 10 monclient(hunting): dump: 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:06.027 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3006903009 0 0) 0x7f7e081d7710 con 0x7f7e0813f710 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 >> v1:172.21.15.22:6790/0 conn(0x7f7e080a98f0 legacy=0x7f7e08137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 >> v1:172.21.15.22:6789/0 conn(0x7f7e081430c0 legacy=0x7f7e081d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 1 monclient: found mon.b 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.031+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e081d84d0 con 0x7f7e0813f710 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7dfd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:30:06.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7dfd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7dfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 monclient: _renew_subs 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e081d41e0 con 0x7f7e0813f710 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 monclient: _renew_subs 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e081d5150 con 0x7f7e0813f710 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 1 librados: init done 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:06.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.032+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e080627b0 con 0x7f7e0813f710 2024-03-25T18:30:06.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.311+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7df80011f0 con 0x7f7e0813f710 2024-03-25T18:30:06.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.311+0000 7f7dfd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:06.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.311+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7df8004cb0 con 0x7f7e0813f710 2024-03-25T18:30:06.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.311+0000 7f7dfd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:06.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.311+0000 7f7dfd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:06.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.311+0000 7f7dfd7fa640 10 monclient: dump: 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:06.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.311+0000 7f7dfeffd640 4 set_mon_vals no callback set 2024-03-25T18:30:06.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.313+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f7df801ad00 con 0x7f7e0813f710 2024-03-25T18:30:06.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.313+0000 7f7e0cb39640 10 monclient: get_auth_request con 0x7f7df406df20 auth_method 0 2024-03-25T18:30:06.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.314+0000 7f7dfd7fa640 10 client.4720.objecter ms_handle_connect 0x7f7df406df20 2024-03-25T18:30:06.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.318+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f7df8054970 con 0x7f7e0813f710 2024-03-25T18:30:06.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.318+0000 7f7dfd7fa640 10 client.4720.objecter ms_dispatch 0x7f7e08000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:06.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.318+0000 7f7dfd7fa640 3 client.4720.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:06.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.318+0000 7f7dfd7fa640 3 client.4720.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:06.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.318+0000 7f7dfd7fa640 20 client.4720.objecter dump_active .. 0 homeless 2024-03-25T18:30:06.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.323+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7df80502b0 con 0x7f7e0813f710 2024-03-25T18:30:06.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.323+0000 7f7dfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:06.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.323+0000 7f7dfd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:06.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.546+0000 7f7e0ed9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:06.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.546+0000 7f7e0ed9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:06.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.546+0000 7f7e0ed9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:06.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.546+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e08074cc0 con 0x7f7e0813f710 2024-03-25T18:30:06.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.574+0000 7f7dfd7fa640 1 -- 172.21.15.22:0/3068338228 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f7df80523b0 con 0x7f7e0813f710 2024-03-25T18:30:06.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.574+0000 7f7dfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:06.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.574+0000 7f7dfd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:06.576 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:30:06.576 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:06.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 10 librados: watch_flush enter 2024-03-25T18:30:06.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 10 librados: watch_flush exit 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 20 client.4720.objecter shutdown clearing up homeless session... 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 10 client.4720.objecter successfully canceled tick 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 >> v1:172.21.15.33:6800/215350958 conn(0x7f7df406df20 legacy=0x7f7df40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 10 monclient: shutdown 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 >> v1:172.21.15.33:6789/0 conn(0x7f7e0813f710 legacy=0x7f7e081d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 shutdown_connections 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.576+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 >> 172.21.15.22:0/3068338228 conn(0x7f7e080b7ea0 msgr2=0x7f7e08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.577+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 shutdown_connections 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.577+0000 7f7e0ed9d640 1 -- 172.21.15.22:0/3068338228 wait complete. 2024-03-25T18:30:06.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:06.577+0000 7f7e0ed9d640 1 librados: shutdown 2024-03-25T18:30:06.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:13.397 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:13.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.510+0000 7f6f05b15640 10 monclient: get_monmap_and_config 2024-03-25T18:30:13.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.511+0000 7f6f05b15640 10 monclient: build_initial_monmap 2024-03-25T18:30:13.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.511+0000 7f6f05b15640 10 monclient: monmap: 2024-03-25T18:30:13.512 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:13.512 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:30:13.512318+0000 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:30:13.512318+0000 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:30:13.513 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.511+0000 7f6f05b15640 1 Processor -- start 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.511+0000 7f6f05b15640 1 -- start start 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.511+0000 7f6f05b15640 10 monclient: init 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0006c380 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0006d670 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 10 monclient(hunting): picked mon.noname-a con 0x7f6f000af8f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0006c9a0 con 0x7f6f000af8f0 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000ada90 con 0x7f6f0006c380 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f000b0d10 con 0x7f6f0006d670 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:13.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.512+0000 7f6f05b15640 10 monclient(hunting): authenticate will time out at 7812.736413s 2024-03-25T18:30:13.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.516+0000 7f6eff7fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f6f0006c380 0x7f6f000b0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:50198/0 (socket says 172.21.15.22:50198) 2024-03-25T18:30:13.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.517+0000 7f6eff7fe640 1 -- 172.21.15.22:0/1327916346 learned_addr learned my addr 172.21.15.22:0/1327916346 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:13.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.530+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754944443 0 0) 0x7f6f000ada90 con 0x7f6f0006c380 2024-03-25T18:30:13.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.530+0000 7f6efe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.530+0000 7f6efe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.530+0000 7f6efe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.530+0000 7f6efe7fc640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:30:13.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.530+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee8003520 con 0x7f6f0006c380 2024-03-25T18:30:13.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.535+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939979349 0 0) 0x7f6f0006c9a0 con 0x7f6f000af8f0 2024-03-25T18:30:13.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.535+0000 7f6efe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.535+0000 7f6efe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.535+0000 7f6efe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.535+0000 7f6efe7fc640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:30:13.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.535+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f000ada90 con 0x7f6f000af8f0 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.538+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706565004 0 0) 0x7f6f000b0d10 con 0x7f6f0006d670 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.538+0000 7f6efe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.538+0000 7f6efe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.538+0000 7f6efe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.538+0000 7f6efe7fc640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:30:13.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.538+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f0006c9a0 con 0x7f6f0006d670 2024-03-25T18:30:13.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 222638133 0 0) 0x7f6ee8003520 con 0x7f6f0006c380 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 >> v1:172.21.15.22:6790/0 conn(0x7f6f0006d670 legacy=0x7f6f000ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 >> v1:172.21.15.22:6789/0 conn(0x7f6f000af8f0 legacy=0x7f6f000aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 1 monclient: found mon.noname-c 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f000add00 con 0x7f6f0006c380 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6ef0001690 con 0x7f6f0006c380 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:13.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6efe7fc640 10 monclient: dump: 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6f05b15640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:30:13.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.539+0000 7f6f05b15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:13.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ef00017f0 con 0x7f6f0006c380 2024-03-25T18:30:13.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6efe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:13.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6efe7fc640 1 -- 172.21.15.22:0/1327916346 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6ef0001b80 con 0x7f6f0006c380 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6efe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6efe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6efe7fc640 10 monclient: dump: 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6f05b15640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:13.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.551+0000 7f6f05b15640 4 set_mon_vals no callback set 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 10 monclient: shutdown 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 1 -- 172.21.15.22:0/1327916346 >> v1:172.21.15.33:6789/0 conn(0x7f6f0006c380 legacy=0x7f6f000b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 1 -- 172.21.15.22:0/1327916346 shutdown_connections 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 1 -- 172.21.15.22:0/1327916346 >> 172.21.15.22:0/1327916346 conn(0x7f6f000b5c00 msgr2=0x7f6f000b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 1 -- 172.21.15.22:0/1327916346 shutdown_connections 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 1 -- 172.21.15.22:0/1327916346 wait complete. 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 10 monclient: build_initial_monmap 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.552+0000 7f6f05b15640 10 monclient: monmap: 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:30:13.556 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 1 librados: starting msgr at 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 1 librados: starting objecter 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 1 Processor -- start 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 1 -- start start 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 1 librados: setting wanted keys 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 1 librados: calling monclient init 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 10 monclient: init 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.553+0000 7f6f05b15640 10 monclient(hunting): picked mon.noname-c con 0x7f6f0006c380 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:13.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.554+0000 7f6f05b15640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0006d670 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.554+0000 7f6f05b15640 10 monclient(hunting): picked mon.noname-a con 0x7f6f000af8f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.554+0000 7f6f05b15640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f00141230 con 0x7f6f000af8f0 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.554+0000 7f6f05b15640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0013ffa0 con 0x7f6f0006d670 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.554+0000 7f6f05b15640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f00140130 con 0x7f6f0006c380 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.554+0000 7f6f05b15640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:13.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.554+0000 7f6f05b15640 10 monclient(hunting): authenticate will time out at 7812.777736s 2024-03-25T18:30:13.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.560+0000 7f6efffff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6f000af8f0 0x7f6f001da7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:36020/0 (socket says 172.21.15.22:36020) 2024-03-25T18:30:13.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.560+0000 7f6efffff640 1 -- 172.21.15.22:0/2053360694 learned_addr learned my addr 172.21.15.22:0/2053360694 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.563+0000 7f6ee7fff640 10 client.?.objecter ms_handle_connect 0x7f6f000af8f0 2024-03-25T18:30:13.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.563+0000 7f6ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:13.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.568+0000 7f6ee7fff640 10 client.?.objecter ms_handle_connect 0x7f6f0006c380 2024-03-25T18:30:13.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.568+0000 7f6ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:13.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62203226 0 0) 0x7f6f00141230 con 0x7f6f000af8f0 2024-03-25T18:30:13.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:30:13.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6eec004eb0 con 0x7f6f000af8f0 2024-03-25T18:30:13.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 10 client.?.objecter ms_handle_connect 0x7f6f0006d670 2024-03-25T18:30:13.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.573+0000 7f6ee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:13.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.574+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244894770 0 0) 0x7f6f00140130 con 0x7f6f0006c380 2024-03-25T18:30:13.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.574+0000 7f6ee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.574+0000 7f6ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.574+0000 7f6ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.574+0000 7f6ee7fff640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:30:13.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.574+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f00141230 con 0x7f6f0006c380 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.587+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140820848 0 0) 0x7f6f0013ffa0 con 0x7f6f0006d670 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.587+0000 7f6ee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.587+0000 7f6ee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.587+0000 7f6ee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.587+0000 7f6ee7fff640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:30:13.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.587+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f00140130 con 0x7f6f0006d670 2024-03-25T18:30:13.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.595+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 922315326 0 0) 0x7f6f00141230 con 0x7f6f0006c380 2024-03-25T18:30:13.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.596+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f0013ffa0 con 0x7f6f0006c380 2024-03-25T18:30:13.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.596+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6ef0001b00 con 0x7f6f0006c380 2024-03-25T18:30:13.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.596+0000 7f6ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.596+0000 7f6ee7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:13.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.596+0000 7f6ee7fff640 10 monclient(hunting): dump: 2024-03-25T18:30:13.597 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.596+0000 7f6ee7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.598+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764257306 0 0) 0x7f6f00140130 con 0x7f6f0006d670 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.598+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f00141230 con 0x7f6f0006d670 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.598+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6ef4001830 con 0x7f6f0006d670 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.598+0000 7f6ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.598+0000 7f6ee7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.598+0000 7f6ee7fff640 10 monclient(hunting): dump: 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:13.599 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:13.600 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.599+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 954164494 0 0) 0x7f6eec004eb0 con 0x7f6f000af8f0 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.599+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f00140130 con 0x7f6f000af8f0 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.599+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6ef8001c90 con 0x7f6f000af8f0 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.599+0000 7f6ee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.599+0000 7f6ee7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.599+0000 7f6ee7fff640 10 monclient(hunting): dump: 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:13.601 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:13.602 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2888833982 0 0) 0x7f6f0013ffa0 con 0x7f6f0006c380 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 >> v1:172.21.15.33:6789/0 conn(0x7f6f0006d670 legacy=0x7f6f0013dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 >> v1:172.21.15.22:6789/0 conn(0x7f6f000af8f0 legacy=0x7f6f001da7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 1 monclient: found mon.c 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00144270 con 0x7f6f0006c380 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 10 monclient: _finish_auth 0 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6ee7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:13.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 5 monclient: authenticate success, global_id 4730 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 10 monclient: _renew_subs 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f00144520 con 0x7f6f0006c380 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 10 monclient: _renew_subs 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0013e8c0 con 0x7f6f0006c380 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.612+0000 7f6f05b15640 1 librados: init done 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.613+0000 7f6f05b15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.613+0000 7f6f05b15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.613+0000 7f6f05b15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:30:13.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.613+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f000af480 con 0x7f6f0006c380 2024-03-25T18:30:13.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.624+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ef00010e0 con 0x7f6f0006c380 2024-03-25T18:30:13.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.624+0000 7f6ee7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:13.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.624+0000 7f6efdffb640 4 set_mon_vals no callback set 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.948+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6ef0004c30 con 0x7f6f0006c380 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.948+0000 7f6ee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.948+0000 7f6ee7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.948+0000 7f6ee7fff640 10 monclient: dump: 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:13.950 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:13.951 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:13.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.949+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f6ef001ac00 con 0x7f6f0006c380 2024-03-25T18:30:13.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.949+0000 7f6efeffd640 10 monclient: get_auth_request con 0x7f6eec06e0f0 auth_method 0 2024-03-25T18:30:13.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.950+0000 7f6ee7fff640 10 client.4730.objecter ms_handle_connect 0x7f6eec06e0f0 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f6ef0054800 con 0x7f6f0006c380 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 10 client.4730.objecter ms_dispatch 0x7f6f00000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 3 client.4730.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 3 client.4730.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 20 client.4730.objecter dump_active .. 0 homeless 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ef0050310 con 0x7f6f0006c380 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:13.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:13.955+0000 7f6ee7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:14.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.197+0000 7f6f05b15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:14.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.197+0000 7f6f05b15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:14.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.197+0000 7f6f05b15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-25T18:30:14.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.197+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f000627b0 con 0x7f6f0006c380 2024-03-25T18:30:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.230+0000 7f6ee7fff640 1 -- 172.21.15.22:0/2053360694 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f6ef0052240 con 0x7f6f0006c380 2024-03-25T18:30:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.230+0000 7f6ee7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.230+0000 7f6ee7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:14.232 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:30:14.233 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:14.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 10 librados: watch_flush enter 2024-03-25T18:30:14.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 10 librados: watch_flush exit 2024-03-25T18:30:14.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 20 client.4730.objecter shutdown clearing up homeless session... 2024-03-25T18:30:14.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 10 client.4730.objecter successfully canceled tick 2024-03-25T18:30:14.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 >> v1:172.21.15.33:6800/215350958 conn(0x7f6eec06e0f0 legacy=0x7f6eec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 10 monclient: shutdown 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 >> v1:172.21.15.22:6790/0 conn(0x7f6f0006c380 legacy=0x7f6f001409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 shutdown_connections 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 >> 172.21.15.22:0/2053360694 conn(0x7f6f000b5c00 msgr2=0x7f6f0014e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 shutdown_connections 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.233+0000 7f6f05b15640 1 -- 172.21.15.22:0/2053360694 wait complete. 2024-03-25T18:30:14.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:14.234+0000 7f6f05b15640 1 librados: shutdown 2024-03-25T18:30:14.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:20.043 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:22.705 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.047 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:22.709 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.071 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:22.688 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.071 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:22.732 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.072 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:22.733 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.072 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:22.690 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.074 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:22.736 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.091 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:22.709 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:20.092 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:22.754 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:21.055 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:21.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.179+0000 7f608e2e4640 10 monclient: get_monmap_and_config 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.179+0000 7f608e2e4640 10 monclient: build_initial_monmap 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.179+0000 7f608e2e4640 10 monclient: monmap: 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:30:21.180726+0000 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:30:21.180726+0000 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:30:21.181 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 1 Processor -- start 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 1 -- start start 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 10 monclient: init 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f608806c380 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f608806d670 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.180+0000 7f608e2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f60880af8f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:21.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.181+0000 7f608e2e4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608806c9a0 con 0x7f60880af8f0 2024-03-25T18:30:21.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.181+0000 7f608e2e4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60880ada90 con 0x7f608806d670 2024-03-25T18:30:21.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.181+0000 7f608e2e4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60880b0d10 con 0x7f608806c380 2024-03-25T18:30:21.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.181+0000 7f608e2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:21.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.181+0000 7f608e2e4640 10 monclient(hunting): authenticate will time out at 7820.404644s 2024-03-25T18:30:21.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.184+0000 7f60877fe640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f608806d670 0x7f60880ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:46678/0 (socket says 172.21.15.22:46678) 2024-03-25T18:30:21.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.184+0000 7f60877fe640 1 -- 172.21.15.22:0/3954735270 learned_addr learned my addr 172.21.15.22:0/3954735270 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:21.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.199+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939965720 0 0) 0x7f60880ada90 con 0x7f608806d670 2024-03-25T18:30:21.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.199+0000 7f6086ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.199+0000 7f6086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.199+0000 7f6086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.199+0000 7f6086ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:30:21.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.199+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6070003520 con 0x7f608806d670 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.200+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984901970 0 0) 0x7f60880b0d10 con 0x7f608806c380 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.200+0000 7f6086ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.200+0000 7f6086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.200+0000 7f6086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.200+0000 7f6086ffd640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:30:21.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.200+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60880ada90 con 0x7f608806c380 2024-03-25T18:30:21.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.210+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929720903 0 0) 0x7f608806c9a0 con 0x7f60880af8f0 2024-03-25T18:30:21.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.210+0000 7f6086ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.210+0000 7f6086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.210+0000 7f6086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.210+0000 7f6086ffd640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:30:21.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.210+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60880b0d10 con 0x7f60880af8f0 2024-03-25T18:30:21.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515176048 0 0) 0x7f60880ada90 con 0x7f608806c380 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 >> v1:172.21.15.33:6789/0 conn(0x7f608806d670 legacy=0x7f60880ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 >> v1:172.21.15.22:6789/0 conn(0x7f60880af8f0 legacy=0x7f60880aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 1 monclient: found mon.noname-b 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60880add00 con 0x7f608806c380 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6078001670 con 0x7f608806c380 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f608e2e4640 5 monclient: authenticate success, global_id 4733 2024-03-25T18:30:21.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f608e2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f6086ffd640 10 monclient: dump: 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.226+0000 7f608e2e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:21.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60780017d0 con 0x7f608806c380 2024-03-25T18:30:21.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f6086ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:21.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f6086ffd640 1 -- 172.21.15.22:0/3954735270 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6078001b60 con 0x7f608806c380 2024-03-25T18:30:21.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f608e2e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:21.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f608e2e4640 4 set_mon_vals no callback set 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f6086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f6086ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f6086ffd640 10 monclient: dump: 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:21.240 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f608e2e4640 10 monclient: shutdown 2024-03-25T18:30:21.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f608e2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:21.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.238+0000 7f608e2e4640 1 -- 172.21.15.22:0/3954735270 >> v1:172.21.15.22:6790/0 conn(0x7f608806c380 legacy=0x7f60880b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.239+0000 7f608e2e4640 1 -- 172.21.15.22:0/3954735270 shutdown_connections 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.239+0000 7f608e2e4640 1 -- 172.21.15.22:0/3954735270 >> 172.21.15.22:0/3954735270 conn(0x7f60880b5c00 msgr2=0x7f60880b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.239+0000 7f608e2e4640 1 -- 172.21.15.22:0/3954735270 shutdown_connections 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.239+0000 7f608e2e4640 1 -- 172.21.15.22:0/3954735270 wait complete. 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.239+0000 7f608e2e4640 10 monclient: build_initial_monmap 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.239+0000 7f608e2e4640 10 monclient: monmap: 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:30:21.243 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.240+0000 7f608e2e4640 1 librados: starting msgr at 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.240+0000 7f608e2e4640 1 librados: starting objecter 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.240+0000 7f608e2e4640 1 Processor -- start 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.240+0000 7f608e2e4640 1 -- start start 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.240+0000 7f608e2e4640 1 librados: setting wanted keys 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.240+0000 7f608e2e4640 1 librados: calling monclient init 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.240+0000 7f608e2e4640 10 monclient: init 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 10 monclient(hunting): picked mon.noname-a con 0x7f608806c380 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 10 monclient(hunting): picked mon.noname-c con 0x7f608806d670 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 10 monclient(hunting): picked mon.noname-b con 0x7f60880af8f0 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088140e60 con 0x7f608806c380 2024-03-25T18:30:21.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088140ff0 con 0x7f60880af8f0 2024-03-25T18:30:21.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088139580 con 0x7f608806d670 2024-03-25T18:30:21.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:21.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.241+0000 7f608e2e4640 10 monclient(hunting): authenticate will time out at 7820.464721s 2024-03-25T18:30:21.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.250+0000 7f608c881640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f60880af8f0 0x7f60881e29f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:46730/0 (socket says 172.21.15.22:46730) 2024-03-25T18:30:21.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.250+0000 7f608c881640 1 -- 172.21.15.22:0/3425611090 learned_addr learned my addr 172.21.15.22:0/3425611090 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:21.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.252+0000 7f6084ff9640 10 client.?.objecter ms_handle_connect 0x7f60880af8f0 2024-03-25T18:30:21.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.252+0000 7f6084ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:21.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.252+0000 7f6084ff9640 10 client.?.objecter ms_handle_connect 0x7f608806d670 2024-03-25T18:30:21.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.252+0000 7f6084ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:21.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.256+0000 7f6084ff9640 10 client.?.objecter ms_handle_connect 0x7f608806c380 2024-03-25T18:30:21.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.256+0000 7f6084ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779995535 0 0) 0x7f6088140ff0 con 0x7f60880af8f0 2024-03-25T18:30:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:30:21.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6074004eb0 con 0x7f60880af8f0 2024-03-25T18:30:21.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350469760 0 0) 0x7f6088139580 con 0x7f608806d670 2024-03-25T18:30:21.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:30:21.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.265+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6088140ff0 con 0x7f608806d670 2024-03-25T18:30:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.271+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859292820 0 0) 0x7f6088140e60 con 0x7f608806c380 2024-03-25T18:30:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.271+0000 7f6084ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:30:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.271+0000 7f6084ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.271+0000 7f6084ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:21.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.271+0000 7f6084ff9640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:30:21.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.271+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6088139580 con 0x7f608806c380 2024-03-25T18:30:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.290+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 361934887 0 0) 0x7f6088140ff0 con 0x7f608806d670 2024-03-25T18:30:21.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.290+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6088140e60 con 0x7f608806d670 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.290+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f607c001950 con 0x7f608806d670 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.290+0000 7f6084ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.290+0000 7f6084ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.290+0000 7f6084ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:21.292 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:21.293 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:21.293 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:21.293 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.290+0000 7f6084ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.292+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1993265447 0 0) 0x7f6074004eb0 con 0x7f60880af8f0 2024-03-25T18:30:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.293+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6088140ff0 con 0x7f60880af8f0 2024-03-25T18:30:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.293+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6080001c70 con 0x7f60880af8f0 2024-03-25T18:30:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.293+0000 7f6084ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.293+0000 7f6084ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.293+0000 7f6084ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:21.295 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.299+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707638212 0 0) 0x7f6088139580 con 0x7f608806c380 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.299+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6074004eb0 con 0x7f608806c380 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.299+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6078001ae0 con 0x7f608806c380 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.299+0000 7f6084ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.300+0000 7f6084ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.300+0000 7f6084ff9640 10 monclient(hunting): dump: 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:21.301 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:21.302 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1939478567 0 0) 0x7f6088140ff0 con 0x7f60880af8f0 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 >> v1:172.21.15.22:6790/0 conn(0x7f608806d670 legacy=0x7f60881da6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 >> v1:172.21.15.22:6789/0 conn(0x7f608806c380 legacy=0x7f6088138e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 1 monclient: found mon.b 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088139710 con 0x7f60880af8f0 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 10 monclient: _finish_auth 0 2024-03-25T18:30:21.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f6084ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 5 monclient: authenticate success, global_id 4732 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 10 monclient: _renew_subs 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6088144b20 con 0x7f60880af8f0 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 10 monclient: _renew_subs 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6088141500 con 0x7f60880af8f0 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.301+0000 7f608e2e4640 1 librados: init done 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.302+0000 7f608e2e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.302+0000 7f608e2e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.302+0000 7f608e2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:21.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.302+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6088139900 con 0x7f60880af8f0 2024-03-25T18:30:21.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.615+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6080003990 con 0x7f60880af8f0 2024-03-25T18:30:21.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.615+0000 7f6084ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:21.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.615+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6080003b80 con 0x7f60880af8f0 2024-03-25T18:30:21.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.615+0000 7f6084ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:21.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.615+0000 7f6084ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:21.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.615+0000 7f6084ff9640 10 monclient: dump: 2024-03-25T18:30:21.617 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:21.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.615+0000 7f60867fc640 4 set_mon_vals no callback set 2024-03-25T18:30:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.617+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f6080003d60 con 0x7f60880af8f0 2024-03-25T18:30:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.617+0000 7f6087fff640 10 monclient: get_auth_request con 0x7f607406e160 auth_method 0 2024-03-25T18:30:21.619 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.617+0000 7f6084ff9640 10 client.4732.objecter ms_handle_connect 0x7f607406e160 2024-03-25T18:30:21.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.621+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f6080054940 con 0x7f60880af8f0 2024-03-25T18:30:21.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.621+0000 7f6084ff9640 10 client.4732.objecter ms_dispatch 0x7f6088000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:21.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.621+0000 7f6084ff9640 3 client.4732.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:21.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.621+0000 7f6084ff9640 3 client.4732.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:21.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.621+0000 7f6084ff9640 20 client.4732.objecter dump_active .. 0 homeless 2024-03-25T18:30:21.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.622+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6080050450 con 0x7f60880af8f0 2024-03-25T18:30:21.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.622+0000 7f6084ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:21.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.622+0000 7f6084ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:21.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.880+0000 7f608e2e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.880+0000 7f608e2e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.880+0000 7f608e2e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:21.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.880+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6088144ec0 con 0x7f60880af8f0 2024-03-25T18:30:21.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.915+0000 7f6084ff9640 1 -- 172.21.15.22:0/3425611090 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f6080052380 con 0x7f60880af8f0 2024-03-25T18:30:21.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.916+0000 7f6084ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:21.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.916+0000 7f6084ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:21.917 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:30:21.918 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:21.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 10 librados: watch_flush enter 2024-03-25T18:30:21.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 10 librados: watch_flush exit 2024-03-25T18:30:21.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 20 client.4732.objecter shutdown clearing up homeless session... 2024-03-25T18:30:21.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 10 client.4732.objecter successfully canceled tick 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 >> v1:172.21.15.33:6800/215350958 conn(0x7f607406e160 legacy=0x7f6074070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 10 monclient: shutdown 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.918+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 >> v1:172.21.15.33:6789/0 conn(0x7f60880af8f0 legacy=0x7f60881e29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.919+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 shutdown_connections 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.919+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 >> 172.21.15.22:0/3425611090 conn(0x7f60880b5c00 msgr2=0x7f60881482f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.919+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 shutdown_connections 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.919+0000 7f608e2e4640 1 -- 172.21.15.22:0/3425611090 wait complete. 2024-03-25T18:30:21.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:21.919+0000 7f608e2e4640 1 librados: shutdown 2024-03-25T18:30:21.932 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:24.436 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:27.097 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.492 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:27.154 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.528 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:27.190 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.539 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:27.157 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.545 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:27.163 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.550 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:27.212 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.555 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:27.217 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.581 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:27.199 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:24.586 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:27.247 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:28.740 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:28.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.852+0000 7f2325a82640 10 monclient: get_monmap_and_config 2024-03-25T18:30:28.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.852+0000 7f2325a82640 10 monclient: build_initial_monmap 2024-03-25T18:30:28.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.852+0000 7f2325a82640 10 monclient: monmap: 2024-03-25T18:30:28.854 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:28.854 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:30:28.854185+0000 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:30:28.854185+0000 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:30:28.855 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:28.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.853+0000 7f2325a82640 1 Processor -- start 2024-03-25T18:30:28.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.853+0000 7f2325a82640 1 -- start start 2024-03-25T18:30:28.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.853+0000 7f2325a82640 10 monclient: init 2024-03-25T18:30:28.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:28.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:30:28.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 10 monclient(hunting): picked mon.noname-a con 0x7f23200a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:28.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 10 monclient(hunting): picked mon.noname-c con 0x7f232013f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 10 monclient(hunting): picked mon.noname-b con 0x7f23201430c0 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2320142520 con 0x7f23200a98f0 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200b28a0 con 0x7f232013f710 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200b2a30 con 0x7f23201430c0 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:28.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.854+0000 7f2325a82640 10 monclient(hunting): authenticate will time out at 7828.078381s 2024-03-25T18:30:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.864+0000 7f231f7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f23200a98f0 0x7f23200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39762/0 (socket says 172.21.15.22:39762) 2024-03-25T18:30:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.864+0000 7f231f7fe640 1 -- 172.21.15.22:0/2760973211 learned_addr learned my addr 172.21.15.22:0/2760973211 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.876+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93693799 0 0) 0x7f2320142520 con 0x7f23200a98f0 2024-03-25T18:30:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.876+0000 7f231e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.876+0000 7f231e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.876+0000 7f231e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.876+0000 7f231e7fc640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:30:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.876+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2308003520 con 0x7f23200a98f0 2024-03-25T18:30:28.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132677203 0 0) 0x7f2308003520 con 0x7f23200a98f0 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 >> v1:172.21.15.22:6790/0 conn(0x7f23201430c0 legacy=0x7f232014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 >> v1:172.21.15.33:6789/0 conn(0x7f232013f710 legacy=0x7f23201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23200b44c0 con 0x7f23200a98f0 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2310001690 con 0x7f23200a98f0 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f231e7fc640 10 monclient: dump: 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f2325a82640 5 monclient: authenticate success, global_id 4722 2024-03-25T18:30:28.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.892+0000 7f2325a82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23100017f0 con 0x7f23200a98f0 2024-03-25T18:30:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f231e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f231e7fc640 1 -- 172.21.15.22:0/2760973211 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2310001c00 con 0x7f23200a98f0 2024-03-25T18:30:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f2325a82640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f2325a82640 4 set_mon_vals no callback set 2024-03-25T18:30:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f231e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f231e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f231e7fc640 10 monclient: dump: 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:28.913 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f2325a82640 10 monclient: shutdown 2024-03-25T18:30:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f2325a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.911+0000 7f2325a82640 1 -- 172.21.15.22:0/2760973211 >> v1:172.21.15.22:6789/0 conn(0x7f23200a98f0 legacy=0x7f23200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.912+0000 7f2325a82640 1 -- 172.21.15.22:0/2760973211 shutdown_connections 2024-03-25T18:30:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.912+0000 7f2325a82640 1 -- 172.21.15.22:0/2760973211 >> 172.21.15.22:0/2760973211 conn(0x7f23200b7ea0 msgr2=0x7f23200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.912+0000 7f2325a82640 1 -- 172.21.15.22:0/2760973211 shutdown_connections 2024-03-25T18:30:28.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.912+0000 7f2325a82640 1 -- 172.21.15.22:0/2760973211 wait complete. 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.912+0000 7f2325a82640 10 monclient: build_initial_monmap 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.913+0000 7f2325a82640 10 monclient: monmap: 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:30:28.915 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:28.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.913+0000 7f2325a82640 1 librados: starting msgr at 2024-03-25T18:30:28.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.913+0000 7f2325a82640 1 librados: starting objecter 2024-03-25T18:30:28.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.913+0000 7f2325a82640 1 Processor -- start 2024-03-25T18:30:28.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 1 -- start start 2024-03-25T18:30:28.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 1 librados: setting wanted keys 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 1 librados: calling monclient init 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient: init 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient(hunting): picked mon.noname-c con 0x7f23200a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient(hunting): picked mon.noname-a con 0x7f232013f710 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient(hunting): picked mon.noname-b con 0x7f23201430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23200abd30 con 0x7f232013f710 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23201d77a0 con 0x7f23201430c0 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23201d51f0 con 0x7f23200a98f0 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:28.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.914+0000 7f2325a82640 10 monclient(hunting): authenticate will time out at 7828.138317s 2024-03-25T18:30:28.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.929+0000 7f231effd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f232013f710 0x7f2320138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39796/0 (socket says 172.21.15.22:39796) 2024-03-25T18:30:28.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.929+0000 7f231effd640 1 -- 172.21.15.22:0/3012263492 learned_addr learned my addr 172.21.15.22:0/3012263492 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:28.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.932+0000 7f2307fff640 10 client.?.objecter ms_handle_connect 0x7f232013f710 2024-03-25T18:30:28.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.932+0000 7f2307fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.934+0000 7f2307fff640 10 client.?.objecter ms_handle_connect 0x7f23201430c0 2024-03-25T18:30:28.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.934+0000 7f2307fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.948+0000 7f2307fff640 10 client.?.objecter ms_handle_connect 0x7f23200a98f0 2024-03-25T18:30:28.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.948+0000 7f2307fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:28.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.954+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441452742 0 0) 0x7f23200abd30 con 0x7f232013f710 2024-03-25T18:30:28.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.954+0000 7f2307fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.954+0000 7f2307fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.954+0000 7f2307fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.954+0000 7f2307fff640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:30:28.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.954+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230c004eb0 con 0x7f232013f710 2024-03-25T18:30:28.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.955+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712508630 0 0) 0x7f23201d51f0 con 0x7f23200a98f0 2024-03-25T18:30:28.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.955+0000 7f2307fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.955+0000 7f2307fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.955+0000 7f2307fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.955+0000 7f2307fff640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:30:28.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.955+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23200abd30 con 0x7f23200a98f0 2024-03-25T18:30:28.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.965+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 61772110 0 0) 0x7f23201d77a0 con 0x7f23201430c0 2024-03-25T18:30:28.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.965+0000 7f2307fff640 10 monclient(hunting): _negotiate 2024-03-25T18:30:28.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.965+0000 7f2307fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:28.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.965+0000 7f2307fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:28.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.965+0000 7f2307fff640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:30:28.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.965+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23201d51f0 con 0x7f23201430c0 2024-03-25T18:30:28.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.976+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658037625 0 0) 0x7f23200abd30 con 0x7f23200a98f0 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.976+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23201d77a0 con 0x7f23200a98f0 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.976+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2310001b40 con 0x7f23200a98f0 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.976+0000 7f2307fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.976+0000 7f2307fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.976+0000 7f2307fff640 10 monclient(hunting): dump: 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:28.978 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:28.979 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:28.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.976+0000 7f2307fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.981+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 397992397 0 0) 0x7f230c004eb0 con 0x7f232013f710 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.981+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23200abd30 con 0x7f232013f710 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.981+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2314001a30 con 0x7f232013f710 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.981+0000 7f2307fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.981+0000 7f2307fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.981+0000 7f2307fff640 10 monclient(hunting): dump: 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.983 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.984 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.984 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:28.984 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:28.984 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:28.984 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:28.984 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.991+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1668619808 0 0) 0x7f23201d51f0 con 0x7f23201430c0 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.991+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f230c004eb0 con 0x7f23201430c0 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.991+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2318001a60 con 0x7f23201430c0 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.991+0000 7f2307fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.991+0000 7f2307fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:28.991+0000 7f2307fff640 10 monclient(hunting): dump: 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:28.993 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:28.994 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2513477012 0 0) 0x7f230c004eb0 con 0x7f23201430c0 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 >> v1:172.21.15.22:6790/0 conn(0x7f23200a98f0 legacy=0x7f23201d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 >> v1:172.21.15.22:6789/0 conn(0x7f232013f710 legacy=0x7f2320138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 10 monclient: _finish_hunting 0 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 1 monclient: found mon.b 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23201d83d0 con 0x7f23201430c0 2024-03-25T18:30:29.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 10 monclient: _finish_auth 0 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 10 monclient: _check_auth_tickets 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2307fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 10 monclient: _renew_subs 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2320138a50 con 0x7f23201430c0 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 10 client.4738.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 10 monclient: _renew_subs 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23201e35c0 con 0x7f23201430c0 2024-03-25T18:30:29.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.009+0000 7f2325a82640 1 librados: init done 2024-03-25T18:30:29.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.010+0000 7f2325a82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:29.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.010+0000 7f2325a82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:29.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.010+0000 7f2325a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:29.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.010+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2320074cc0 con 0x7f23201430c0 2024-03-25T18:30:29.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.340+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2318001f90 con 0x7f23201430c0 2024-03-25T18:30:29.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.340+0000 7f2307fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.340+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f2318003b10 con 0x7f23201430c0 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.340+0000 7f2307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.340+0000 7f2307fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.340+0000 7f2307fff640 10 monclient: dump: 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:29.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.340+0000 7f231dffb640 4 set_mon_vals no callback set 2024-03-25T18:30:29.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.341+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f2318019ae0 con 0x7f23201430c0 2024-03-25T18:30:29.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.341+0000 7f231f7fe640 10 monclient: get_auth_request con 0x7f230c06e110 auth_method 0 2024-03-25T18:30:29.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.342+0000 7f2307fff640 10 client.4738.objecter ms_handle_connect 0x7f230c06e110 2024-03-25T18:30:29.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f2318053780 con 0x7f23201430c0 2024-03-25T18:30:29.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 10 client.4738.objecter ms_dispatch 0x7f2320000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:29.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 3 client.4738.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:29.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 3 client.4738.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:29.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 20 client.4738.objecter dump_active .. 0 homeless 2024-03-25T18:30:29.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f231804f0c0 con 0x7f23201430c0 2024-03-25T18:30:29.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:29.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.346+0000 7f2307fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.597+0000 7f2325a82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:29.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.597+0000 7f2325a82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:29.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.597+0000 7f2325a82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:29.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.597+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23201d8560 con 0x7f23201430c0 2024-03-25T18:30:29.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.619+0000 7f2307fff640 1 -- 172.21.15.22:0/3012263492 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f23180511c0 con 0x7f23201430c0 2024-03-25T18:30:29.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.619+0000 7f2307fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:29.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.619+0000 7f2307fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:29.621 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:30:29.621 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:29.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 10 librados: watch_flush enter 2024-03-25T18:30:29.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 10 librados: watch_flush exit 2024-03-25T18:30:29.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 20 client.4738.objecter shutdown clearing up homeless session... 2024-03-25T18:30:29.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 10 client.4738.objecter successfully canceled tick 2024-03-25T18:30:29.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 >> v1:172.21.15.33:6800/215350958 conn(0x7f230c06e110 legacy=0x7f230c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:29.623 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 10 monclient: shutdown 2024-03-25T18:30:29.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:29.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 >> v1:172.21.15.33:6789/0 conn(0x7f23201430c0 legacy=0x7f23201e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:29.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 shutdown_connections 2024-03-25T18:30:29.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.621+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 >> 172.21.15.22:0/3012263492 conn(0x7f23200b7ea0 msgr2=0x7f2320074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:29.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.622+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 shutdown_connections 2024-03-25T18:30:29.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.622+0000 7f2325a82640 1 -- 172.21.15.22:0/3012263492 wait complete. 2024-03-25T18:30:29.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:29.622+0000 7f2325a82640 1 librados: shutdown 2024-03-25T18:30:29.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:36.442 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.560+0000 7f6fc0b8c640 10 monclient: get_monmap_and_config 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.560+0000 7f6fc0b8c640 10 monclient: build_initial_monmap 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.560+0000 7f6fc0b8c640 10 monclient: monmap: 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:30:36.560942+0000 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:30:36.560942+0000 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:30:36.561 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:36.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.561+0000 7f6fc0b8c640 1 Processor -- start 2024-03-25T18:30:36.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.561+0000 7f6fc0b8c640 1 -- start start 2024-03-25T18:30:36.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.561+0000 7f6fc0b8c640 10 monclient: init 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 10 monclient(hunting): picked mon.noname-a con 0x7f6fbc0a98f0 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 10 monclient(hunting): picked mon.noname-b con 0x7f6fbc13f710 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 10 monclient(hunting): picked mon.noname-c con 0x7f6fbc1430c0 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc142520 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc0b28a0 con 0x7f6fbc1430c0 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc0b2a30 con 0x7f6fbc13f710 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:36.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.562+0000 7f6fc0b8c640 10 monclient(hunting): authenticate will time out at 7835.785037s 2024-03-25T18:30:36.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.576+0000 7f6fba59c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6fbc0a98f0 0x7f6fbc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:56176/0 (socket says 172.21.15.22:56176) 2024-03-25T18:30:36.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.576+0000 7f6fba59c640 1 -- 172.21.15.22:0/4192186137 learned_addr learned my addr 172.21.15.22:0/4192186137 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:36.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.595+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961552791 0 0) 0x7f6fbc142520 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.595+0000 7f6fb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.595+0000 7f6fb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.595+0000 7f6fb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.595+0000 7f6fb9d9b640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:30:36.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.595+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa8003520 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595670358 0 0) 0x7f6fbc0b2a30 con 0x7f6fbc13f710 2024-03-25T18:30:36.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): my global_id is 4745 2024-03-25T18:30:36.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc142520 con 0x7f6fbc13f710 2024-03-25T18:30:36.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323581517 0 0) 0x7f6fbc0b28a0 con 0x7f6fbc1430c0 2024-03-25T18:30:36.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:30:36.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.601+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc0b2a30 con 0x7f6fbc1430c0 2024-03-25T18:30:36.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129639365 0 0) 0x7f6fbc142520 con 0x7f6fbc13f710 2024-03-25T18:30:36.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 >> v1:172.21.15.33:6789/0 conn(0x7f6fbc1430c0 legacy=0x7f6fbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 >> v1:172.21.15.22:6789/0 conn(0x7f6fbc0a98f0 legacy=0x7f6fbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 1 monclient: found mon.noname-b 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc0b44c0 con 0x7f6fbc13f710 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fa4001670 con 0x7f6fbc13f710 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.612+0000 7f6fb9d9b640 10 monclient: dump: 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:36.614 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.613+0000 7f6fc0b8c640 5 monclient: authenticate success, global_id 4745 2024-03-25T18:30:36.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.613+0000 7f6fc0b8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fa40017d0 con 0x7f6fbc13f710 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fb9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fb9d9b640 1 -- 172.21.15.22:0/4192186137 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fa4001be0 con 0x7f6fbc13f710 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fc0b8c640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fc0b8c640 4 set_mon_vals no callback set 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fb9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fb9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:36.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.633+0000 7f6fb9d9b640 10 monclient: dump: 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:36.635 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:36.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.634+0000 7f6fc0b8c640 10 monclient: shutdown 2024-03-25T18:30:36.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.634+0000 7f6fc0b8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:36.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.634+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/4192186137 >> v1:172.21.15.22:6790/0 conn(0x7f6fbc13f710 legacy=0x7f6fbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.634+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/4192186137 shutdown_connections 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.634+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/4192186137 >> 172.21.15.22:0/4192186137 conn(0x7f6fbc0b7ea0 msgr2=0x7f6fbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.634+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/4192186137 shutdown_connections 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.634+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/4192186137 wait complete. 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.635+0000 7f6fc0b8c640 10 monclient: build_initial_monmap 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.635+0000 7f6fc0b8c640 10 monclient: monmap: 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:30:36.638 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.635+0000 7f6fc0b8c640 1 librados: starting msgr at 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.635+0000 7f6fc0b8c640 1 librados: starting objecter 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 1 Processor -- start 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 1 -- start start 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 1 librados: setting wanted keys 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 1 librados: calling monclient init 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 10 monclient: init 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 10 monclient(hunting): picked mon.noname-c con 0x7f6fbc0a98f0 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 10 monclient(hunting): picked mon.noname-b con 0x7f6fbc13f710 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 10 monclient(hunting): picked mon.noname-a con 0x7f6fbc1430c0 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc136fa0 con 0x7f6fbc1430c0 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.636+0000 7f6fc0b8c640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc13b780 con 0x7f6fbc13f710 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.637+0000 7f6fc0b8c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc1d7ba0 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.637+0000 7f6fc0b8c640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:36.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.637+0000 7f6fc0b8c640 10 monclient(hunting): authenticate will time out at 7835.859644s 2024-03-25T18:30:36.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.653+0000 7f6fb3fff640 1 --1- >> v1:172.21.15.33:6789/0 conn(0x7f6fbc13f710 0x7f6fbc1d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.33:6789/0 says I am v1:172.21.15.22:40604/0 (socket says 172.21.15.22:40604) 2024-03-25T18:30:36.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.653+0000 7f6fb3fff640 1 -- 172.21.15.22:0/1036420154 learned_addr learned my addr 172.21.15.22:0/1036420154 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:36.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.656+0000 7f6fb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6fbc13f710 2024-03-25T18:30:36.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.656+0000 7f6fb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:36.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.660+0000 7f6fb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6fbc0a98f0 2024-03-25T18:30:36.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.660+0000 7f6fb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:36.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.664+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496568522 0 0) 0x7f6fbc13b780 con 0x7f6fbc13f710 2024-03-25T18:30:36.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.664+0000 7f6fb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.664+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.664+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.664+0000 7f6fb2ffd640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:30:36.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.664+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f9c004eb0 con 0x7f6fbc13f710 2024-03-25T18:30:36.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.674+0000 7f6fb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6fbc1430c0 2024-03-25T18:30:36.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.674+0000 7f6fb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:36.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.675+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55986482 0 0) 0x7f6fbc1d7ba0 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.675+0000 7f6fb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:30:36.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.675+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:36.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.675+0000 7f6fb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:36.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.675+0000 7f6fb2ffd640 10 monclient(hunting): my global_id is 4748 2024-03-25T18:30:36.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.675+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc13b780 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.688+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 742981088 0 0) 0x7f6fbc13b780 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.688+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fbc1d7ba0 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.688+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fac001950 con 0x7f6fbc0a98f0 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.688+0000 7f6fb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.688+0000 7f6fb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.688+0000 7f6fb2ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.689 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.688+0000 7f6fb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:36.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.690+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2571533792 0 0) 0x7f6f9c004eb0 con 0x7f6fbc13f710 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.690+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fbc13b780 con 0x7f6fbc13f710 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.690+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fa4001670 con 0x7f6fbc13f710 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.690+0000 7f6fb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.690+0000 7f6fb2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.690+0000 7f6fb2ffd640 10 monclient(hunting): dump: 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:36.691 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:36.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1733432008 0 0) 0x7f6fbc13b780 con 0x7f6fbc13f710 2024-03-25T18:30:36.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 >> v1:172.21.15.22:6790/0 conn(0x7f6fbc0a98f0 legacy=0x7f6fbc136890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 >> v1:172.21.15.22:6789/0 conn(0x7f6fbc1430c0 legacy=0x7f6fbc1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:36.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:30:36.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 1 monclient: found mon.b 2024-03-25T18:30:36.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc137990 con 0x7f6fbc13f710 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fc0b8c640 5 monclient: authenticate success, global_id 4744 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fc0b8c640 10 monclient: _renew_subs 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fc0b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.706+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc137c40 con 0x7f6fbc13f710 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.707+0000 7f6fc0b8c640 10 client.4744.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.707+0000 7f6fc0b8c640 10 monclient: _renew_subs 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.707+0000 7f6fc0b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.707+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc139860 con 0x7f6fbc13f710 2024-03-25T18:30:36.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.707+0000 7f6fc0b8c640 1 librados: init done 2024-03-25T18:30:36.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.708+0000 7f6fc0b8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:36.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.708+0000 7f6fc0b8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:36.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.708+0000 7f6fc0b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:36.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:36.708+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fbc0627b0 con 0x7f6fbc13f710 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.035+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fa40038a0 con 0x7f6fbc13f710 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.035+0000 7f6fb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.035+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f6fa4003cf0 con 0x7f6fbc13f710 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.035+0000 7f6fb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.035+0000 7f6fb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.035+0000 7f6fb2ffd640 10 monclient: dump: 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:37.036 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:37.037 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:37.037 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:37.037 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:37.037 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:37.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.035+0000 7f6fb959a640 4 set_mon_vals no callback set 2024-03-25T18:30:37.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.036+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f6fa4003f10 con 0x7f6fbc13f710 2024-03-25T18:30:37.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.037+0000 7f6fba59c640 10 monclient: get_auth_request con 0x7f6f9c06de80 auth_method 0 2024-03-25T18:30:37.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.037+0000 7f6fb2ffd640 10 client.4744.objecter ms_handle_connect 0x7f6f9c06de80 2024-03-25T18:30:37.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.041+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f6fa40549c0 con 0x7f6fbc13f710 2024-03-25T18:30:37.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.041+0000 7f6fb2ffd640 10 client.4744.objecter ms_dispatch 0x7f6fbc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:37.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.041+0000 7f6fb2ffd640 3 client.4744.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:37.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.041+0000 7f6fb2ffd640 3 client.4744.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:37.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.041+0000 7f6fb2ffd640 20 client.4744.objecter dump_active .. 0 homeless 2024-03-25T18:30:37.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.042+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fa40504d0 con 0x7f6fbc13f710 2024-03-25T18:30:37.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.042+0000 7f6fb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:37.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.042+0000 7f6fb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:37.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.275+0000 7f6fc0b8c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:37.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.275+0000 7f6fc0b8c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:37.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.275+0000 7f6fc0b8c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:37.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.275+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fbc069f40 con 0x7f6fbc13f710 2024-03-25T18:30:37.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.380+0000 7f6fb2ffd640 1 -- 172.21.15.22:0/1036420154 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f6fa4052400 con 0x7f6fbc13f710 2024-03-25T18:30:37.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.380+0000 7f6fb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:37.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.380+0000 7f6fb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:37.381 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:30:37.382 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 10 librados: watch_flush enter 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 10 librados: watch_flush exit 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 20 client.4744.objecter shutdown clearing up homeless session... 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 10 client.4744.objecter successfully canceled tick 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 >> v1:172.21.15.33:6800/215350958 conn(0x7f6f9c06de80 legacy=0x7f6f9c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 10 monclient: shutdown 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 >> v1:172.21.15.33:6789/0 conn(0x7f6fbc13f710 legacy=0x7f6fbc1d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 shutdown_connections 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 >> 172.21.15.22:0/1036420154 conn(0x7f6fbc0b7ea0 msgr2=0x7f6fbc14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 shutdown_connections 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 1 -- 172.21.15.22:0/1036420154 wait complete. 2024-03-25T18:30:37.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:37.383+0000 7f6fc0b8c640 1 librados: shutdown 2024-03-25T18:30:37.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:44.202 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:30:44.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.316+0000 7f70ad438640 10 monclient: get_monmap_and_config 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.316+0000 7f70ad438640 10 monclient: build_initial_monmap 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.316+0000 7f70ad438640 10 monclient: monmap: 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:30:44.316367+0000 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:30:44.316367+0000 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-c 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-25T18:30:44.317 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 1 Processor -- start 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 1 -- start start 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient: init 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient(hunting): picked mon.noname-b con 0x7f70a80b4d60 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient(hunting): picked mon.noname-c con 0x7f70a80b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient(hunting): picked mon.noname-a con 0x7f70a814be90 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80af940 con 0x7f70a814be90 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80b0a70 con 0x7f70a80b1730 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80b0d50 con 0x7f70a80b4d60 2024-03-25T18:30:44.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:44.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.317+0000 7f70ad438640 10 monclient(hunting): authenticate will time out at 7843.540554s 2024-03-25T18:30:44.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.324+0000 7f70a6ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f70a80b4d60 0x7f70a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60704/0 (socket says 172.21.15.22:60704) 2024-03-25T18:30:44.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.324+0000 7f70a6ffd640 1 -- 172.21.15.22:0/625660436 learned_addr learned my addr 172.21.15.22:0/625660436 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:44.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.336+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102644494 0 0) 0x7f70a80b0a70 con 0x7f70a80b1730 2024-03-25T18:30:44.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.336+0000 7f70a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.336+0000 7f70a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.336+0000 7f70a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.336+0000 7f70a67fc640 10 monclient(hunting): my global_id is 4747 2024-03-25T18:30:44.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.336+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7094003520 con 0x7f70a80b1730 2024-03-25T18:30:44.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.341+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798363462 0 0) 0x7f70a80b0d50 con 0x7f70a80b4d60 2024-03-25T18:30:44.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.341+0000 7f70a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.341+0000 7f70a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.341+0000 7f70a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.341+0000 7f70a67fc640 10 monclient(hunting): my global_id is 4751 2024-03-25T18:30:44.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.341+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a80b0a70 con 0x7f70a80b4d60 2024-03-25T18:30:44.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 51419991 0 0) 0x7f7094003520 con 0x7f70a80b1730 2024-03-25T18:30:44.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 >> v1:172.21.15.22:6790/0 conn(0x7f70a80b4d60 legacy=0x7f70a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 >> v1:172.21.15.22:6789/0 conn(0x7f70a814be90 legacy=0x7f70a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.348 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 1 monclient: found mon.noname-c 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.33:6789/0 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a80b1030 con 0x7f70a80b1730 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 10 monclient: _finish_auth 0 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7090001690 con 0x7f70a80b1730 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70a67fc640 10 monclient: dump: 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.349 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70ad438640 5 monclient: authenticate success, global_id 4747 2024-03-25T18:30:44.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.348+0000 7f70ad438640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:30:44.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 <== mon.1 v1:172.21.15.33:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70900017f0 con 0x7f70a80b1730 2024-03-25T18:30:44.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:44.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70a67fc640 1 -- 172.21.15.22:0/625660436 <== mon.1 v1:172.21.15.33:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7090001c00 con 0x7f70a80b1730 2024-03-25T18:30:44.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70ad438640 10 monclient: get_monmap_and_config success 2024-03-25T18:30:44.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70ad438640 4 set_mon_vals no callback set 2024-03-25T18:30:44.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70a67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70a67fc640 10 monclient: dump: 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:44.353 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70ad438640 10 monclient: shutdown 2024-03-25T18:30:44.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70ad438640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.352+0000 7f70ad438640 1 -- 172.21.15.22:0/625660436 >> v1:172.21.15.33:6789/0 conn(0x7f70a80b1730 legacy=0x7f70a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.353+0000 7f70ad438640 1 -- 172.21.15.22:0/625660436 shutdown_connections 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.353+0000 7f70ad438640 1 -- 172.21.15.22:0/625660436 >> 172.21.15.22:0/625660436 conn(0x7f70a80617a0 msgr2=0x7f70a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.353+0000 7f70ad438640 1 -- 172.21.15.22:0/625660436 shutdown_connections 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.353+0000 7f70ad438640 1 -- 172.21.15.22:0/625660436 wait complete. 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 10 monclient: build_initial_monmap 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 10 monclient: monmap: 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-25T18:30:44.355 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.noname-b 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 1 librados: starting msgr at 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 1 librados: starting objecter 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 1 Processor -- start 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 1 -- start start 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 1 librados: setting wanted keys 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 1 librados: calling monclient init 2024-03-25T18:30:44.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.354+0000 7f70ad438640 10 monclient: init 2024-03-25T18:30:44.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 10 monclient: _reopen_session rank -1 2024-03-25T18:30:44.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:30:44.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 10 monclient(hunting): picked mon.noname-b con 0x7f70a80b1730 addr v1:172.21.15.33:6789/0 2024-03-25T18:30:44.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 10 monclient(hunting): picked mon.noname-a con 0x7f70a80b4d60 addr v1:172.21.15.22:6789/0 2024-03-25T18:30:44.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 10 monclient(hunting): picked mon.noname-c con 0x7f70a814be90 addr v1:172.21.15.22:6790/0 2024-03-25T18:30:44.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8135810 con 0x7f70a80b4d60 2024-03-25T18:30:44.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 1 -- --> v1:172.21.15.33:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8144c10 con 0x7f70a80b1730 2024-03-25T18:30:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a8144ef0 con 0x7f70a814be90 2024-03-25T18:30:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 10 monclient(hunting): _renew_subs 2024-03-25T18:30:44.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.355+0000 7f70ad438640 10 monclient(hunting): authenticate will time out at 7843.578053s 2024-03-25T18:30:44.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.367+0000 7f709ffff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f70a80b4d60 0x7f70a81349f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44502/0 (socket says 172.21.15.22:44502) 2024-03-25T18:30:44.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.367+0000 7f709ffff640 1 -- 172.21.15.22:0/3854374590 learned_addr learned my addr 172.21.15.22:0/3854374590 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-25T18:30:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.369+0000 7f709f7fe640 10 client.?.objecter ms_handle_connect 0x7f70a80b4d60 2024-03-25T18:30:44.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.369+0000 7f709f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:44.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.377+0000 7f709f7fe640 10 client.?.objecter ms_handle_connect 0x7f70a814be90 2024-03-25T18:30:44.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.377+0000 7f709f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.379+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798735641 0 0) 0x7f70a8135810 con 0x7f70a80b4d60 2024-03-25T18:30:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.379+0000 7f709f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.379+0000 7f709f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.379+0000 7f709f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.379+0000 7f709f7fe640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:30:44.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.379+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7088004eb0 con 0x7f70a80b4d60 2024-03-25T18:30:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.380+0000 7f709f7fe640 10 client.?.objecter ms_handle_connect 0x7f70a80b1730 2024-03-25T18:30:44.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.380+0000 7f709f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:30:44.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.384+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111544947 0 0) 0x7f70a8144ef0 con 0x7f70a814be90 2024-03-25T18:30:44.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.384+0000 7f709f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.384+0000 7f709f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.384+0000 7f709f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.384+0000 7f709f7fe640 10 monclient(hunting): my global_id is 4754 2024-03-25T18:30:44.385 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.384+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a8135810 con 0x7f70a814be90 2024-03-25T18:30:44.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.391+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746596770 0 0) 0x7f70a8144c10 con 0x7f70a80b1730 2024-03-25T18:30:44.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.391+0000 7f709f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:30:44.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.391+0000 7f709f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:30:44.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.391+0000 7f709f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:30:44.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.391+0000 7f709f7fe640 10 monclient(hunting): my global_id is 4750 2024-03-25T18:30:44.392 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.391+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.33:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a8144ef0 con 0x7f70a80b1730 2024-03-25T18:30:44.413 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.413+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2413049691 0 0) 0x7f70a8144ef0 con 0x7f70a80b1730 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.413+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.33:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a8144c10 con 0x7f70a80b1730 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.413+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7098001930 con 0x7f70a80b1730 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.413+0000 7f709f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.413+0000 7f709f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.413+0000 7f709f7fe640 10 monclient(hunting): dump: 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:44.414 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:44.415 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:44.415 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:44.415 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.413+0000 7f709f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:30:44.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1242756846 0 0) 0x7f70a8135810 con 0x7f70a814be90 2024-03-25T18:30:44.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a8144ef0 con 0x7f70a814be90 2024-03-25T18:30:44.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f70a0001b00 con 0x7f70a814be90 2024-03-25T18:30:44.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:30:44.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 10 monclient(hunting): dump: 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3510393372 0 0) 0x7f7088004eb0 con 0x7f70a80b4d60 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70a8135810 con 0x7f70a80b4d60 2024-03-25T18:30:44.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7090001690 con 0x7f70a80b4d60 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.424+0000 7f709f7fe640 10 monclient(hunting): dump: 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:44.427 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4064163243 0 0) 0x7f70a8144c10 con 0x7f70a80b1730 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 >> v1:172.21.15.22:6790/0 conn(0x7f70a814be90 legacy=0x7f70a8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 >> v1:172.21.15.22:6789/0 conn(0x7f70a80b4d60 legacy=0x7f70a81349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 1 monclient: found mon.b 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:44.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.33:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a8140c10 con 0x7f70a80b1730 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f709f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 5 monclient: authenticate success, global_id 4750 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 10 monclient: _renew_subs 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.33:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a8137d50 con 0x7f70a80b1730 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 10 client.4750.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 10 monclient: _renew_subs 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.33:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a81407d0 con 0x7f70a80b1730 2024-03-25T18:30:44.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.439+0000 7f70ad438640 1 librados: init done 2024-03-25T18:30:44.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.440+0000 7f70ad438640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:44.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.440+0000 7f70ad438640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:44.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.440+0000 7f70ad438640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:44.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.440+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a8141e30 con 0x7f70a80b1730 2024-03-25T18:30:44.507 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:47.169 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.560 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:47.221 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.590 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:11:47.252 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.615 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:47.277 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.622 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:47.284 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.653 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:11:47.315 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.783 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:47.401 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.786 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:47.404 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.804+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70980047e0 con 0x7f70a80b1730 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.804+0000 7f709f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.804+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3222232396 0 0) 0x7f7098001d80 con 0x7f70a80b1730 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.804+0000 7f709f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.804+0000 7f709f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.804+0000 7f709f7fe640 10 monclient: dump: 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:fsid 3e36bd18-e4e0-4ff2-83ef-b1243b472434 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-25T18:18:26.397533+0000 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 3 2024-03-25T18:30:44.805 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-25T18:30:44.806 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.33:6789/0 mon.b 2024-03-25T18:30:44.806 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-25T18:30:44.806 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-25T18:30:44.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.804+0000 7f70a5ffb640 4 set_mon_vals no callback set 2024-03-25T18:30:44.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.805+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1456844227 0 0) 0x7f709801a9e0 con 0x7f70a80b1730 2024-03-25T18:30:44.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.806+0000 7f709ffff640 10 monclient: get_auth_request con 0x7f708806e0f0 auth_method 0 2024-03-25T18:30:44.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.806+0000 7f709f7fe640 10 client.4750.objecter ms_handle_connect 0x7f708806e0f0 2024-03-25T18:30:44.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.812+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3894535888 0 0) 0x7f7098054750 con 0x7f70a80b1730 2024-03-25T18:30:44.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.812+0000 7f709f7fe640 10 client.4750.objecter ms_dispatch 0x7f70a8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:30:44.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.812+0000 7f709f7fe640 3 client.4750.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:30:44.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.812+0000 7f709f7fe640 3 client.4750.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:30:44.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.812+0000 7f709f7fe640 20 client.4750.objecter dump_active .. 0 homeless 2024-03-25T18:30:44.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.813+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7098050090 con 0x7f70a80b1730 2024-03-25T18:30:44.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.813+0000 7f709f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:30:44.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:44.813+0000 7f709f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:30:44.825 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:11:47.443 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:30:45.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.061+0000 7f70ad438640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:45.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.061+0000 7f70ad438640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:45.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.061+0000 7f70ad438640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.33:6789/0 2024-03-25T18:30:45.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.061+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 --> v1:172.21.15.33:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70a8145080 con 0x7f70a80b1730 2024-03-25T18:30:45.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.106+0000 7f709f7fe640 1 -- 172.21.15.22:0/3854374590 <== mon.1 v1:172.21.15.33:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1660713027) 0x7f7098052190 con 0x7f70a80b1730 2024-03-25T18:30:45.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.106+0000 7f709f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:30:45.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.106+0000 7f709f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:30:45.107 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-25T18:30:45.107 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"3e36bd18-e4e0-4ff2-83ef-b1243b472434","created":"2024-03-25T18:19:14.779713+0000","modified":"2024-03-25T18:29:19.063989+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:29:19.063989+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6253bf71-82c4-48c3-86bf-f3cbff91fb52","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eb34859b-b0d5-410a-ad0a-3fc1cfbb6e70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1268cb30-a412-4d09-8729-a87a96e89819","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"fe29129e-9772-43fd-9f2a-4251290662ca","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"bf2a5aa7-1c5e-4935-bd80-4b115bce0ff0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ab490014-db2a-47a2-92f8-308d207490c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8a5817bc-3728-4133-879e-37557b37c520","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ddfa42ca-42f4-4851-a399-5d591d922824","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:30:45.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 10 librados: watch_flush enter 2024-03-25T18:30:45.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 10 librados: watch_flush exit 2024-03-25T18:30:45.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 20 client.4750.objecter shutdown clearing up homeless session... 2024-03-25T18:30:45.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 10 client.4750.objecter successfully canceled tick 2024-03-25T18:30:45.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 >> v1:172.21.15.33:6800/215350958 conn(0x7f708806e0f0 legacy=0x7f70880705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 10 monclient: shutdown 2024-03-25T18:30:45.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:30:45.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 >> v1:172.21.15.33:6789/0 conn(0x7f70a80b1730 legacy=0x7f70a81d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:30:45.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 shutdown_connections 2024-03-25T18:30:45.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 >> 172.21.15.22:0/3854374590 conn(0x7f70a80617a0 msgr2=0x7f70a80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:30:45.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 shutdown_connections 2024-03-25T18:30:45.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 1 -- 172.21.15.22:0/3854374590 wait complete. 2024-03-25T18:30:45.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-25T18:30:45.108+0000 7f70ad438640 1 librados: shutdown 2024-03-25T18:30:45.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:30:45.120 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:30:45.122 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:30:45.122 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:30:45.122 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:30:47.037 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:11:26.670 35450== 2024-03-25T18:30:47.037 INFO:tasks.ceph.osd.0.smithi022.stderr:==00:00:11:26.670 35450== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:30:47.068 INFO:tasks.ceph.osd.0.smithi022.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:30:51.223 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:30:51.224 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:30:51.224 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:30:51.225 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:30:51.225 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:30:53.239 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:11:32.865 35452== 2024-03-25T18:30:53.239 INFO:tasks.ceph.osd.1.smithi022.stderr:==00:00:11:32.865 35452== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:30:53.281 INFO:tasks.ceph.osd.1.smithi022.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:30:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:30:57.327 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:30:57.328 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:30:57.328 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:30:57.328 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:30:59.324 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:11:38.968 35449== 2024-03-25T18:30:59.324 INFO:tasks.ceph.osd.2.smithi022.stderr:==00:00:11:38.968 35449== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:30:59.365 INFO:tasks.ceph.osd.2.smithi022.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:03.430 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:31:03.431 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:31:03.431 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:31:03.431 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:04.563 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:07.225 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.567 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:07.229 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.596 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:07.258 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.620 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:07.238 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.623 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:07.285 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.625 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:07.243 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.629 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:07.291 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.648 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:07.266 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:04.655 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:07.317 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:05.151 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:11:44.779 35451== 2024-03-25T18:31:05.151 INFO:tasks.ceph.osd.3.smithi022.stderr:==00:00:11:44.779 35451== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:05.186 INFO:tasks.ceph.osd.3.smithi022.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:09.534 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:31:09.534 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:31:09.534 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:31:09.535 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:11.593 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:11:51.236 35191== 2024-03-25T18:31:11.593 INFO:tasks.ceph.osd.4.smithi033.stderr:==00:00:11:51.236 35191== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:11.631 INFO:tasks.ceph.osd.4.smithi033.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:15.636 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:15.637 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:31:15.637 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:31:15.637 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:31:15.638 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:17.614 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:11:57.256 35192== 2024-03-25T18:31:17.614 INFO:tasks.ceph.osd.5.smithi033.stderr:==00:00:11:57.256 35192== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:17.652 INFO:tasks.ceph.osd.5.smithi033.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:21.740 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:31:21.741 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:31:21.742 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:31:21.742 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:23.433 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:12:03.073 35193== 2024-03-25T18:31:23.434 INFO:tasks.ceph.osd.6.smithi033.stderr:==00:00:12:03.074 35193== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:23.484 INFO:tasks.ceph.osd.6.smithi033.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:24.557 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:27.219 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.604 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:27.222 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.605 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:27.266 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.608 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:27.226 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.620 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:27.282 35134== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.625 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:27.242 34646== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.651 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:27.313 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.654 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:27.316 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:24.690 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:27.352 35138== Warning: unimplemented fcntl command: 1036 2024-03-25T18:31:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:27.844 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:31:27.844 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:31:27.845 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:31:27.845 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:29.463 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:12:09.098 35194== 2024-03-25T18:31:29.464 INFO:tasks.ceph.osd.7.smithi033.stderr:==00:00:12:09.098 35194== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:29.494 INFO:tasks.ceph.osd.7.smithi033.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:33.947 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:31:33.947 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:31:33.947 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:31:33.948 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:31:33.948 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:33.975 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:31:33.975 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:31:33.975 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:33.976 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:31:33.975+0000 7f0ef80d3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34644) UID: 0 2024-03-25T18:31:33.976 INFO:tasks.ceph.mgr.x.smithi033.stderr:2024-03-25T18:31:33.975+0000 7f0ef80d3640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:31:34.006 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:31:34.006 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:31:34.006 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:31:34.006 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:34.031 INFO:tasks.ceph.mon.a.smithi022.stderr:2024-03-25T18:31:34.018+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35103) UID: 0 2024-03-25T18:31:34.032 INFO:tasks.ceph.mon.a.smithi022.stderr:2024-03-25T18:31:34.020+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:31:34.620 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:37.281 35134== 2024-03-25T18:31:34.620 INFO:tasks.ceph.mon.a.smithi022.stderr:==00:00:12:37.281 35134== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:34.629 INFO:tasks.ceph.mon.a.smithi022.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:40.108 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:31:40.109 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:31:40.109 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:31:40.109 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:40.125 INFO:tasks.ceph.mon.c.smithi022.stderr:2024-03-25T18:31:40.112+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35105) UID: 0 2024-03-25T18:31:40.126 INFO:tasks.ceph.mon.c.smithi022.stderr:2024-03-25T18:31:40.114+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:31:40.725 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:43.386 35138== 2024-03-25T18:31:40.725 INFO:tasks.ceph.mon.c.smithi022.stderr:==00:00:12:43.386 35138== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:40.735 INFO:tasks.ceph.mon.c.smithi022.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:46.211 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:46.211 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi022 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:31:46.212 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:31:46.212 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:31:46.212 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:31:46.230 INFO:tasks.ceph.mon.b.smithi033.stderr:2024-03-25T18:31:46.215+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34642) UID: 0 2024-03-25T18:31:46.231 INFO:tasks.ceph.mon.b.smithi033.stderr:2024-03-25T18:31:46.217+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:31:46.814 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:49.432 34646== 2024-03-25T18:31:46.815 INFO:tasks.ceph.mon.b.smithi033.stderr:==00:00:12:49.432 34646== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:31:46.824 INFO:tasks.ceph.mon.b.smithi033.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:31:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:31:52.315 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:31:52.315 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:31:52.315 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:31:52.316 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:31:52.366 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:31:52.366 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:31:52.630 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:31:52.631 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:31:52.697 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:31:52.697 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:31:52.754 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:31:52.754 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:31:52.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:31:52.812 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:31:52.971 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:31:52.972 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:31:53.027 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:31:53.027 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:31:53.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:31:53.078 DEBUG:teuthology.orchestra.run.smithi033:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:31:53.136 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:31:53.138 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/data/mon.a.tgz 2024-03-25T18:31:53.138 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-25T18:31:53.153 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.iDrtAV7yOk 2024-03-25T18:31:53.153 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iDrtAV7yOk 2024-03-25T18:31:53.250 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.iDrtAV7yOk 2024-03-25T18:31:53.334 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.iDrtAV7yOk is 134KB 2024-03-25T18:31:53.385 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.iDrtAV7yOk 2024-03-25T18:31:53.399 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/data/mon.c.tgz 2024-03-25T18:31:53.399 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-25T18:31:53.453 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.i6YlLIwmhz 2024-03-25T18:31:53.454 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.i6YlLIwmhz 2024-03-25T18:31:53.543 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.i6YlLIwmhz 2024-03-25T18:31:53.627 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.i6YlLIwmhz is 134KB 2024-03-25T18:31:53.677 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.i6YlLIwmhz 2024-03-25T18:31:53.692 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/data/mon.b.tgz 2024-03-25T18:31:53.692 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-03-25T18:31:53.710 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.nQPkTq77od 2024-03-25T18:31:53.710 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.nQPkTq77od 2024-03-25T18:31:53.816 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.nQPkTq77od 2024-03-25T18:31:53.903 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.nQPkTq77od is 134KB 2024-03-25T18:31:53.953 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.nQPkTq77od 2024-03-25T18:31:53.968 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:31:53.968 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:31:53.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:31:54.352 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:31:54.596 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:31:54.598 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/remote/smithi022/crash 2024-03-25T18:31:54.599 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:31:54.627 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/remote/smithi033/crash 2024-03-25T18:31:54.628 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:31:54.654 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:31:54.655 DEBUG:teuthology.orchestra.run.smithi022:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:31:54.671 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34961.log 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.35001.log 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34961.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35001.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34961.log.gz 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35001.log.gz 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:31:54.744 INFO:teuthology.orchestra.run.smithi033.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34841.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35041.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34841.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34801.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34841.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35041.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34801.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.4.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.34801.log.gz 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34881.log 2024-03-25T18:31:54.745 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.34881.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34881.log.gz 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35092.log 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34921.log 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/ceph-client.admin.35092.log: /var/log/ceph/ceph-client.admin.34921.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35092.log.gz 2024-03-25T18:31:54.746 INFO:teuthology.orchestra.run.smithi033.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34921.log.gz 2024-03-25T18:31:54.754 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2024-03-25T18:31:54.754 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41990.log 2024-03-25T18:31:54.754 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:31:54.754 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39767.log: /var/log/ceph/ceph-client.admin.41990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40508.log 2024-03-25T18:31:54.754 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2024-03-25T18:31:54.754 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41717.log 2024-03-25T18:31:54.754 INFO:teuthology.orchestra.run.smithi022.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41990.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39650.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40508.log: /var/log/ceph/ceph-client.admin.41717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42265.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42068.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40508.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41717.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.42265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39063.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39650.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.42068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41522.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.42265.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39063.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.42068.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40235.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39063.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41522.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40235.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41522.log.gz 2024-03-25T18:31:54.755 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40235.log.gz 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41288.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41171.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38982.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39845.log 2024-03-25T18:31:54.756 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41288.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39806.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41171.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40664.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38982.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39806.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39845.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41405.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9%/var/log/ceph/ceph-client.admin.40664.log: -- replaced with /var/log/ceph/ceph-client.admin.39806.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41249.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42187.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40664.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41405.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.42187.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41249.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41483.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39219.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42187.log.gz 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-03-25T18:31:54.757 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3%/var/log/ceph/ceph-client.admin.39219.log: -- replaced with /var/log/ceph/ceph-client.admin.41483.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39219.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40820.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39728.log: gzip -5 --verbose -- 86.7% /var/log/ceph/ceph-client.admin.40625.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42109.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40898.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.42109.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42029.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40625.log.gz 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40196.log 2024-03-25T18:31:54.758 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42109.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.42029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41600.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40898.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40196.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42029.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40586.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40781.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40196.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40586.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41600.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40430.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40469.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40586.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40001.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40781.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40469.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40430.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41015.log 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40469.log.gz 2024-03-25T18:31:54.759 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39689.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40001.log.gz 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41015.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41015.log.gz 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41795.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39689.log.gz 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41093.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41795.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gzgzip 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39180.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41561.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41093.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41795.log.gz 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41366.log 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41561.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41093.log.gz 2024-03-25T18:31:54.760 INFO:teuthology.orchestra.run.smithi022.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39180.log.gz 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35475.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39571.log/var/log/ceph/ceph-client.admin.41366.log: 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41561.log.gz 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.35475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41366.log.gz 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39571.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35475.log.gz 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39962.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40976.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39571.log.gz 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41873.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.3.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39962.log.gz 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38777.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35631.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41873.log: /var/log/ceph/ceph-client.admin.38777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39493.log 2024-03-25T18:31:54.761 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41873.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.35631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38777.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39022.log 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35631.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40742.log 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39493.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39022.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42226.log 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39022.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41327.log 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.42226.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40742.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41327.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42226.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39336.log 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-mgr.y.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41327.log.gz 2024-03-25T18:31:54.762 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42148.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39141.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.42148.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39336.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39297.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39141.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42148.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39297.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39141.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39258.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39297.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35145.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40157.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41756.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.35145.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37577.log 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41210.log 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35145.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.37577.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41756.log.gz 2024-03-25T18:31:54.763 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41210.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37577.log.gz 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39923.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39532.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.0.log: gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39611.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41912.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39611.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39532.log.gz 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39414.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:gzip 87.0% -5 --verbose -- /var/log/ceph/ceph-client.admin.41054.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39611.log.gz 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41912.log: /var/log/ceph/ceph-client.admin.39414.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41912.log.gz 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39414.log.gz 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40274.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36525.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41132.log 2024-03-25T18:31:54.764 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41054.log.gz 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39454.log 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40274.log.gz/var/log/ceph/ceph-client.admin.36525.log: 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41951.log 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39454.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36525.log.gz 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41132.log.gz 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39454.log.gz 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr: 93.9%/var/log/ceph/ceph-client.admin.41951.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41678.log 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40859.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41951.log.gz 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29291.log 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41678.log: /var/log/ceph/ceph.tmp-client.admin.29291.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29291.log.gz 2024-03-25T18:31:54.765 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41678.log.gz 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40391.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40313.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40391.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41834.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gz 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40391.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39102.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40313.log: /var/log/ceph/ceph-client.admin.41834.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40313.log.gz 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41834.log.gz 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39375.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35448.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39102.log: /var/log/ceph/ceph-client.admin.39375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40352.log 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.35448.log: 87.0% 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39102.log.gz 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39375.log.gz 2024-03-25T18:31:54.766 INFO:teuthology.orchestra.run.smithi022.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35448.log.gz 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40703.log 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41639.log 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40352.log: /var/log/ceph/ceph-client.admin.40703.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41639.log: 87.0% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40352.log.gz 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40703.log.gz 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41639.log.gz 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40547.log 2024-03-25T18:31:54.767 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.40547.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40547.log.gz 2024-03-25T18:31:54.815 INFO:teuthology.orchestra.run.smithi033.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:31:55.534 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:31:55.621 INFO:teuthology.orchestra.run.smithi033.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:31:55.621 INFO:teuthology.orchestra.run.smithi033.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:31:55.655 INFO:teuthology.orchestra.run.smithi033.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:31:55.806 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:31:55.889 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:31:55.907 INFO:teuthology.orchestra.run.smithi033.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:31:55.915 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:31:59.896 INFO:teuthology.orchestra.run.smithi022.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:32:00.044 INFO:teuthology.orchestra.run.smithi022.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:32:00.046 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-25T18:32:00.046 INFO:teuthology.orchestra.run.smithi022.stderr:real 0m5.362s 2024-03-25T18:32:00.046 INFO:teuthology.orchestra.run.smithi022.stderr:user 0m14.402s 2024-03-25T18:32:00.046 INFO:teuthology.orchestra.run.smithi022.stderr:sys 0m0.503s 2024-03-25T18:32:00.154 INFO:teuthology.orchestra.run.smithi033.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:32:00.156 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-03-25T18:32:00.156 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m5.442s 2024-03-25T18:32:00.156 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m9.098s 2024-03-25T18:32:00.157 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.318s 2024-03-25T18:32:00.157 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:32:00.157 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/remote/smithi022/log 2024-03-25T18:32:00.158 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:32:02.432 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/remote/smithi033/log 2024-03-25T18:32:02.433 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:32:03.869 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:32:04.083 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=451f7b52ab9444ed8b7c318979dbed42 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:32:04.088 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:32:04.095 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:32:04.104 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:32:04.104 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:32:04.118 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:32:04.198 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:32:04.206 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:32:04.207 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:32:04.209 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:32:04.221 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:32:04.226 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:32:04.226 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-25T18:32:04.226 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 57 +690us[ +660us] +/- 70ms 2024-03-25T18:32:04.226 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -1106us[-1136us] +/- 55ms 2024-03-25T18:32:04.226 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +839us[ +809us] +/- 75ms 2024-03-25T18:32:04.227 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:32:04.227 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:32:04.230 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:32:04.230 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-03-25T18:32:04.230 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +674us[ +674us] +/- 70ms 2024-03-25T18:32:04.230 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -1125us[-1154us] +/- 55ms 2024-03-25T18:32:04.231 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +813us[ +813us] +/- 75ms 2024-03-25T18:32:04.231 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:32:04.231 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:32:04.256 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:32:04.257 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:32:04.265 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:32:04.334 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:32:04.407 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:32:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:04.437 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:32:04.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:04.469 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:32:04.477 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:32:04.485 INFO:teuthology.task.internal:Duration was 7564.174279 seconds 2024-03-25T18:32:04.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:32:04.493 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:32:04.493 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:32:04.496 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:32:04.536 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:32:04.554 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:32:04.927 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:32:04.927 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi022.front.sepia.ceph.com 2024-03-25T18:32:04.927 DEBUG:teuthology.orchestra.run.smithi022:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:32:04.987 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-03-25T18:32:04.987 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:32:05.014 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:32:05.014 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:32:05.030 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:32:05.086 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:32:05.086 DEBUG:teuthology.orchestra.run.smithi022:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:32:05.097 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:32:05.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:32:05.243 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:32:05.243 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:32:05.270 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:32:05.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:32:05.305 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:05.312 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:05.335 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2024-03-25T18:32:05.361 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-03-25T18:32:05.431 DEBUG:teuthology.orchestra.run.smithi022:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:05.445 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:32:05.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:32:05.485 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:32:05.495 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:32:05.496 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/remote/smithi022 2024-03-25T18:32:05.496 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:32:05.531 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275/remote/smithi033 2024-03-25T18:32:05.532 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:32:05.566 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:32:05.567 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:32:05.569 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:32:05.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:32:05.627 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:32:05.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:32:05.639 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:32:05.640 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:32:05.642 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:32:05.654 INFO:teuthology.orchestra.run.smithi022.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:32 /home/ubuntu/cephtest 2024-03-25T18:32:05.670 INFO:teuthology.orchestra.run.smithi033.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:32 /home/ubuntu/cephtest 2024-03-25T18:32:05.671 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:32:05.680 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:32:05.737 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:32:05.764 DEBUG:teuthology.nuke:shortname: smithi022 2024-03-25T18:32:05.764 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:32:05.793 DEBUG:teuthology.nuke:shortname: smithi033 2024-03-25T18:32:05.793 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:32:05.797 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:17:15.246194', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAOqIqfEQZt615P3doFwq9RpYUjamDcNMxl4wqu0jpH8yYxlrfRlZ9eQhZSGpvnHoa8mNuD8n7zg2fH01SeUZWE='} 2024-03-25T18:32:05.818 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621275', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:17:15.247707', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCi6NsqnuXO4iVWJhs8L6iaW9zhmGjeZxY8mKhUJZvmHnPsroUzqgUn1tmOoUIKd0cSSaEEofFq8BfyG5kZ8ksI='} 2024-03-25T18:32:05.853 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-25T18:32:05.853 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:32:05.878 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-25T18:32:05.879 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:05.893 INFO:teuthology.orchestra.console.smithi033:Power off 2024-03-25T18:32:05.894 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:32:05.903 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-25T18:32:05.915 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2024-03-25T18:32:05.915 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:05.941 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-25T18:32:09.904 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:09.942 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:10.014 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-25T18:32:10.050 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-03-25T18:32:14.017 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:14.051 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:32:14.128 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-25T18:32:14.161 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2024-03-25T18:32:14.229 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-25T18:32:14.261 INFO:teuthology.orchestra.console.smithi033:Power off completed 2024-03-25T18:32:14.366 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7564.17427945137 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=451f7b52ab9444ed8b7c318979dbed42 status: fail success: false 2024-03-25T18:32:14.366 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:32:14.437 INFO:teuthology.run:FAIL